Oct 31 14:45:13.231738 ip-10-0-131-60 systemd[1]: Starting Kubernetes Kubelet... Oct 31 14:45:13.602411 ip-10-0-131-60 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:13.602411 ip-10-0-131-60 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 14:45:13.602411 ip-10-0-131-60 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:13.602411 ip-10-0-131-60 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:13.605851 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605772 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.605851 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605851 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.605851 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605855 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605860 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605865 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605868 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605871 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605874 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605877 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605879 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605889 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605892 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605896 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605900 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605903 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605905 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605908 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605911 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605913 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605916 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605918 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605921 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.605948 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605924 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605926 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605929 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605931 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605934 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605937 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605940 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605942 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605945 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605947 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605950 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605953 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605956 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605958 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605961 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605963 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605966 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605968 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605971 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.606418 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605973 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605975 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605978 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605981 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605983 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605986 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605988 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605990 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605993 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605995 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.605997 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606000 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606003 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606005 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606007 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606010 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606012 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606015 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606017 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606020 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.606887 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606023 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606025 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606028 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606030 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606032 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606035 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606038 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606040 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606043 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606046 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606049 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606051 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606053 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606056 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606058 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606064 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606067 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606069 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606072 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606075 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.607348 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606077 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606080 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606082 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606084 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606087 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606090 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606093 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606095 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606098 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606100 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606103 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606105 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606108 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606111 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606113 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606116 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606119 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606121 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606124 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.607859 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606126 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606129 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606131 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606134 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606137 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606140 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606916 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606925 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606928 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606931 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606934 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606937 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606939 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606942 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606944 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606948 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606952 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606956 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606959 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.608303 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606962 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606965 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606968 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606970 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606973 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606976 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606979 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606982 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606984 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606987 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606990 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606992 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606995 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.606997 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607000 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607002 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607004 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607007 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607009 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607012 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.608828 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607014 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607017 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607019 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607023 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607025 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607027 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607030 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607033 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607035 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607038 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607040 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607043 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607045 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607048 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607050 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607052 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607055 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607058 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607060 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607063 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.609511 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607066 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607069 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607071 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607073 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607076 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607078 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607081 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607083 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607086 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607088 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607090 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607093 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607096 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607100 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607103 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607106 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607109 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607112 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607114 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.610200 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607117 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607119 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607121 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607124 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607126 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607129 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607131 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607133 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607136 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607138 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607141 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607143 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607146 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607149 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607152 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607154 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607157 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607159 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607162 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.610730 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607164 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607167 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607169 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607171 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607174 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607176 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607179 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607182 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607184 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607187 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607189 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607192 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607194 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607197 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607199 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607685 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607690 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607693 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607695 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607698 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.611192 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607700 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607703 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607705 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607708 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607710 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607713 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607716 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607718 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607721 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607724 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607726 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607729 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607731 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607733 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607736 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607738 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607740 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607743 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607746 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.611690 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607748 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607751 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607754 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607756 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607759 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607762 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607764 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607767 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607769 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607772 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607774 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607777 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607779 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607782 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607784 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607787 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607790 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607792 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607795 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607797 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.612143 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607799 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607802 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607804 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607807 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607809 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607812 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607814 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607816 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607819 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607821 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607824 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607827 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607829 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607832 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607834 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607837 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607840 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607842 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607845 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607847 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.612648 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607850 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607852 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607854 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607858 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607861 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607864 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607867 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607870 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607872 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607875 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607877 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607880 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607882 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607884 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607887 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607889 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607893 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607897 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607900 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.613166 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607903 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607905 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607908 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607911 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607917 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607919 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607922 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607925 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607927 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607930 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607932 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607935 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607938 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607941 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607943 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607946 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607948 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607951 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607953 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607956 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.613644 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607958 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607960 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.607963 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608901 2570 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608911 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608917 2570 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608921 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608926 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608929 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608933 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608937 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608940 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608943 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608946 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608949 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608953 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608956 2570 flags.go:64] FLAG: --cgroup-root="" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.608959 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610302 2570 flags.go:64] FLAG: --client-ca-file="" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610318 2570 flags.go:64] FLAG: --cloud-provider="external" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610322 2570 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610345 2570 flags.go:64] FLAG: --cluster-domain="" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610348 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 14:45:13.614124 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610352 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610358 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610362 2570 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610371 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610375 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610378 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610382 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610385 2570 flags.go:64] FLAG: --contention-profiling="false" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610388 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610392 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610395 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610399 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610406 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610409 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610412 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610416 2570 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610419 2570 flags.go:64] FLAG: --enable-server="true" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610422 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610427 2570 flags.go:64] FLAG: --event-burst="100" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610431 2570 flags.go:64] FLAG: --event-qps="50" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610436 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610439 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610442 2570 flags.go:64] FLAG: --eviction-hard="" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610447 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 14:45:13.614676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610450 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610453 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610457 2570 flags.go:64] FLAG: --eviction-soft="" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610460 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610464 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610486 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610489 2570 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610492 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610495 2570 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610498 2570 flags.go:64] FLAG: --feature-gates="" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610503 2570 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610506 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610510 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610515 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610519 2570 flags.go:64] FLAG: --healthz-port="10248" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610522 2570 flags.go:64] FLAG: --help="false" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610525 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610529 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610532 2570 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610535 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610538 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610542 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610547 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 14:45:13.615285 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610551 2570 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610554 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610557 2570 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610560 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610563 2570 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610566 2570 flags.go:64] FLAG: --kube-reserved="" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610569 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610574 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610577 2570 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610580 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610583 2570 flags.go:64] FLAG: --lock-file="" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610588 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610591 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610595 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610601 2570 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610604 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610610 2570 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610613 2570 flags.go:64] FLAG: --logging-format="text" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610632 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610637 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610641 2570 flags.go:64] FLAG: --manifest-url="" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610644 2570 flags.go:64] FLAG: --manifest-url-header="" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610649 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610653 2570 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610660 2570 flags.go:64] FLAG: --max-pods="110" Oct 31 14:45:13.615866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610664 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610667 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610670 2570 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610673 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610676 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610679 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610683 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610694 2570 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610698 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610701 2570 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610704 2570 flags.go:64] FLAG: --pod-cidr="" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610707 2570 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610710 2570 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610713 2570 flags.go:64] FLAG: --pods-per-core="0" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610716 2570 flags.go:64] FLAG: --port="10250" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610722 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610725 2570 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-06e1420c05f26bdac" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610728 2570 flags.go:64] FLAG: --qos-reserved="" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610732 2570 flags.go:64] FLAG: --read-only-port="10255" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610736 2570 flags.go:64] FLAG: --register-node="true" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610739 2570 flags.go:64] FLAG: --register-with-taints="" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610743 2570 flags.go:64] FLAG: --registry-burst="10" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610746 2570 flags.go:64] FLAG: --registry-qps="5" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610751 2570 flags.go:64] FLAG: --reserved-cpus="" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610754 2570 flags.go:64] FLAG: --reserved-memory="" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610758 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 14:45:13.616495 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610761 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610764 2570 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610768 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610771 2570 flags.go:64] FLAG: --runonce="false" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610774 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610779 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610782 2570 flags.go:64] FLAG: --seccomp-default="false" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610785 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610789 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610792 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610795 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610798 2570 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610801 2570 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610805 2570 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610811 2570 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610814 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610817 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610820 2570 flags.go:64] FLAG: --system-cgroups="" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610822 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610829 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610832 2570 flags.go:64] FLAG: --tls-cert-file="" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610837 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610842 2570 flags.go:64] FLAG: --tls-min-version="" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610844 2570 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610848 2570 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 14:45:13.617128 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610851 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610854 2570 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610857 2570 flags.go:64] FLAG: --v="2" Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610862 2570 flags.go:64] FLAG: --version="false" Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610869 2570 flags.go:64] FLAG: --vmodule="" Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610873 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.610876 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611129 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611133 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611136 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611143 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611147 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611150 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611152 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611155 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611158 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611161 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611164 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611166 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611169 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611172 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611174 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.617742 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611177 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611182 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611184 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611187 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611189 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611192 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611195 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611197 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611200 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611204 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611207 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611210 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611212 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611215 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611220 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611223 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611226 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611228 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611231 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.618247 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611233 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611236 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611239 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611242 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611245 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611247 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611250 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611255 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611258 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611260 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611263 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611266 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611268 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611271 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611273 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611276 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611279 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611282 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611284 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611289 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.618722 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611292 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611294 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611298 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611300 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611303 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611306 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611309 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611312 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611314 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611317 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611319 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611322 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611327 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611330 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611332 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611335 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611338 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611340 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611344 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.619219 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611348 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611352 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611355 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611359 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611363 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611369 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611372 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611374 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611377 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611379 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611382 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611385 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611387 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611390 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611392 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611396 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611398 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611401 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.619686 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611406 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611408 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611411 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611414 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611416 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611419 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611421 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611424 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611427 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611429 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611432 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611435 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611438 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611443 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611445 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.611497 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:13.620123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611569 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611576 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611578 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611584 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611587 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611589 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611592 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611595 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611597 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611600 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611603 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611605 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611608 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611610 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611614 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611619 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611622 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611624 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611627 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611629 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.620587 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611632 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611635 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611637 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611640 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611642 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611645 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611647 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611650 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611656 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611658 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611661 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611663 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611666 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611670 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611673 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611675 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611678 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611681 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611683 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611686 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.621095 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611691 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611694 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611697 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611699 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611702 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611705 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611708 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611712 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611714 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611717 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611719 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611722 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611727 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611729 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611732 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611734 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611737 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611739 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611742 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.621581 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611745 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611747 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611750 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611753 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611755 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611758 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611764 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611766 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611769 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611771 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611774 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611776 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611780 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611784 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611787 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611790 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611794 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611798 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611803 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.622031 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611806 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611809 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611812 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611814 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611817 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611820 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611823 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611826 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611828 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611831 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611833 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611836 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611841 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611844 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611846 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611849 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611852 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611854 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611857 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611862 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.622496 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611864 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611867 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611870 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611872 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611877 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611880 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611882 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.611885 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.619346 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.619363 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.619376 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.619381 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619437 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619442 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619445 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619449 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619453 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619456 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619459 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.622958 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619462 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619481 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619486 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619491 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619496 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619499 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619502 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619505 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619508 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619511 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619514 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619516 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619519 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619521 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619524 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619526 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619529 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619531 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619533 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.623409 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619536 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619538 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619541 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619544 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619547 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619549 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619553 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619556 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619558 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619560 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619563 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619566 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619568 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619570 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619573 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619575 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619578 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619580 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619582 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.623869 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619585 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619587 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619590 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619592 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619595 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619597 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619599 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619602 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619604 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619607 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619610 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619612 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619615 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619617 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619620 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619623 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619625 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619628 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619630 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619633 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.624316 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619635 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619638 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619641 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619643 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619645 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619648 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619650 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619653 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619655 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619657 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619660 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619662 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619665 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619667 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619670 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619673 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619677 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619680 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.624951 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619682 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619685 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619688 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619691 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619693 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619696 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619699 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619701 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619704 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619707 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619709 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619712 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619715 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619718 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619720 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619723 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619725 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619727 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619730 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619732 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.625382 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619735 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619738 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619740 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.619767 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619910 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619915 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619918 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619922 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619925 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619927 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619930 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619932 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619934 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619937 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619940 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619942 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.625867 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619945 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619947 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619949 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619952 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619954 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619957 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619959 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619963 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619965 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619968 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619970 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619972 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619975 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619977 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619980 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619982 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619984 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619987 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619990 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.626294 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619995 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.619999 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620002 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620005 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620007 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620010 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620012 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620015 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620017 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620020 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620022 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620025 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620027 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620030 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620032 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620034 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620037 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620039 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620042 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620044 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.626753 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620047 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620050 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620052 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620055 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620057 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620060 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620062 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620064 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620067 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620069 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620071 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620074 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620076 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620079 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620081 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620084 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620086 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620089 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620091 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.627299 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620093 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620096 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620098 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620101 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620104 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620106 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620108 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620111 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620114 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620116 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620118 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620123 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620125 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620128 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620131 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620134 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620136 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620139 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620141 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620144 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.627776 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620146 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620148 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620151 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620153 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620156 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620158 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620161 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620164 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620166 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620169 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620171 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620174 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620176 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620178 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620181 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.628257 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620185 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.620207 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620238 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620242 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620244 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620247 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620250 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620252 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620255 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620259 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620262 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620264 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620267 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620270 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620273 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620275 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:13.628667 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620278 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620281 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620283 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620286 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620288 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620291 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620293 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620296 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620298 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620301 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620303 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620306 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620308 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620311 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620313 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620316 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620318 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620321 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620323 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620326 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:13.629080 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620328 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620330 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620333 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620335 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620338 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620341 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620344 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620346 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620348 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620351 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620354 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620357 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620359 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620362 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620364 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620368 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620372 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620375 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620377 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:13.629551 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620380 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620383 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620386 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620388 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620390 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620393 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620395 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620398 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620400 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620403 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620405 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620408 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620410 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620413 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620415 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620418 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620420 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620422 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620427 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620429 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:13.629992 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620432 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620434 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620437 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620439 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620442 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620445 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620447 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620451 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620454 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620457 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620461 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620463 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620479 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620482 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620485 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620488 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620492 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620494 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620497 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:13.630533 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620499 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620502 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620504 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620507 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620509 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620512 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620514 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620524 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620527 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620530 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620532 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620536 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620540 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:13.620542 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.620602 2570 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.623027 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.623996 2570 server.go:1014] "Starting client certificate rotation" Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.624091 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 14:45:13.630968 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.624125 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 14:45:13.631437 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.626111 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:13.631437 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.627218 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:13.636202 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.636186 2570 log.go:25] "Validated CRI v1 runtime API" Oct 31 14:45:13.641817 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.641796 2570 log.go:25] "Validated CRI v1 image API" Oct 31 14:45:13.643135 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.643110 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 14:45:13.645735 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.645716 2570 fs.go:135] Filesystem UUIDs: map[0985ecc2-f783-4eb3-ae78-4fe5692cec0c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d329977-038f-4bbc-9c40-4e9b2741fa3b:/dev/nvme0n1p4] Oct 31 14:45:13.645783 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.645735 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 14:45:13.651877 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.651741 2570 manager.go:217] Machine: {Timestamp:2026-10-31 14:45:13.650310937 +0000 UTC m=+0.322994693 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098709 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a59a41a61da2f39bc909af4a42d2 SystemUUID:ec24a59a-41a6-1da2-f39b-c909af4a42d2 BootID:4fcd75dc-20b9-494b-a326-7c8b3322a412 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:79:bb:d1:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:79:bb:d1:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:80:57:04:39:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 14:45:13.652503 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.652493 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 14:45:13.652603 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.652590 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 14:45:13.654422 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.654400 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 14:45:13.654578 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.654423 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-60.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 14:45:13.654627 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.654581 2570 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 14:45:13.654627 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.654590 2570 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 14:45:13.654681 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.654633 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:13.654681 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.654649 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:13.655868 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.655858 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 14:45:13.660341 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.660330 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 14:45:13.660458 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.660449 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 14:45:13.661274 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.661264 2570 kubelet.go:493] "Attempting to sync node with API server" Oct 31 14:45:13.661307 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.661279 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 14:45:13.661307 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.661289 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 14:45:13.661307 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.661298 2570 kubelet.go:405] "Adding apiserver pod source" Oct 31 14:45:13.661307 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.661307 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 14:45:13.661650 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.661638 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:13.663945 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.663927 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 14:45:13.670539 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.670527 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:13.671450 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671438 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 14:45:13.671494 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671483 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 14:45:13.671639 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671628 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 14:45:13.671671 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671644 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 14:45:13.671671 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671651 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 14:45:13.671671 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671657 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 14:45:13.671671 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671663 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 14:45:13.671671 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.671670 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 14:45:13.672391 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.672383 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 14:45:13.672422 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.672392 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 14:45:13.672422 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.672399 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 14:45:13.672422 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.672405 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 14:45:13.672422 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.672419 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 14:45:13.672641 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.672429 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 14:45:13.673794 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.673784 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 14:45:13.673825 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.673796 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 14:45:13.677619 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.677606 2570 server.go:1263] "Started kubelet" Oct 31 14:45:13.677702 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.677660 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 14:45:13.677926 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.677864 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 14:45:13.678009 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.677964 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 14:45:13.678562 ip-10-0-131-60 systemd[1]: Started Kubernetes Kubelet. Oct 31 14:45:13.679559 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.679432 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 14:45:13.680363 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.680344 2570 server.go:317] "Adding debug handlers to kubelet server" Oct 31 14:45:13.684211 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.684194 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 14:45:13.686517 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.686498 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 14:45:13.687112 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.687097 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 14:45:13.687688 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.687666 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" Oct 31 14:45:13.688240 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.688221 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 14:45:13.688309 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.688244 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 14:45:13.688309 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.688261 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 14:45:13.688398 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.688390 2570 reconstruct.go:97] "Volume reconstruction finished" Oct 31 14:45:13.688442 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.688398 2570 reconciler.go:29] "Reconciler: start to sync state" Oct 31 14:45:13.688685 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.688665 2570 factory.go:55] Registering systemd factory Oct 31 14:45:13.688754 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.688691 2570 factory.go:223] Registration of the systemd container factory successfully Oct 31 14:45:13.689025 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.689009 2570 factory.go:153] Registering CRI-O factory Oct 31 14:45:13.689079 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.689028 2570 factory.go:223] Registration of the crio container factory successfully Oct 31 14:45:13.689136 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.689125 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 14:45:13.689191 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.689145 2570 factory.go:103] Registering Raw factory Oct 31 14:45:13.689191 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.689160 2570 manager.go:1199] Started watching for new ooms in manager Oct 31 14:45:13.692334 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.692315 2570 manager.go:319] Starting recovery of all containers Oct 31 14:45:13.702435 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.702418 2570 manager.go:324] Recovery completed Oct 31 14:45:13.703863 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.703845 2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 31 14:45:13.706433 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.706422 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:13.707833 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.707798 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:13.707892 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.707839 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:13.707892 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.707851 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:13.708329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.708303 2570 cpu_manager.go:226] "Starting" policy="none" Oct 31 14:45:13.708329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.708316 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 14:45:13.708329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.708320 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:13.708561 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.708336 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 14:45:13.709027 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.709012 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:13.709090 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.709035 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:13.709090 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.709045 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:13.710621 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.710610 2570 policy_none.go:50] "Start" Oct 31 14:45:13.710660 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.710628 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 14:45:13.710660 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.710639 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 14:45:13.712630 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.712619 2570 policy_none.go:44] "Start" Oct 31 14:45:13.748181 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.748047 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 14:45:13.752481 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.752452 2570 manager.go:341] "Starting Device Plugin manager" Oct 31 14:45:13.752551 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.752503 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 14:45:13.752551 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.752523 2570 server.go:87] "Starting device plugin registration server" Oct 31 14:45:13.752736 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.752726 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 14:45:13.752768 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.752738 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 14:45:13.752860 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.752840 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 14:45:13.752956 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.752929 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 14:45:13.752956 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.752938 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 14:45:13.753458 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.753439 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 14:45:13.753544 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.753510 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" Oct 31 14:45:13.819956 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.819929 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-60.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 14:45:13.821720 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.819968 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-60.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 14:45:13.821720 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.819929 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-60.us-west-1.compute.internal.18e3a478685d8272 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-60.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-60.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 14:45:13.677578866 +0000 UTC m=+0.350262619,LastTimestamp:2026-10-31 14:45:13.677578866 +0000 UTC m=+0.350262619,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-60.us-west-1.compute.internal,}" Oct 31 14:45:13.840123 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.840096 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 14:45:13.841391 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.841373 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 14:45:13.841391 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.841390 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 14:45:13.841547 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.841405 2570 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 14:45:13.841547 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.841434 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 14:45:13.853742 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.853691 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:13.854602 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.854569 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:13.854685 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.854623 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:13.854685 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.854650 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:13.854685 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.854681 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:13.897457 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.897416 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 14:45:13.897562 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.897522 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8wl77" Oct 31 14:45:13.942045 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.941999 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal"] Oct 31 14:45:13.942111 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.942088 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:13.943011 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.942997 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:13.943083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.943024 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:13.943083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.943035 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:13.945089 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945078 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:13.945236 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945223 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:13.945271 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945254 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:13.945741 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945724 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:13.945802 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945724 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:13.945802 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945756 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:13.945802 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945767 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:13.945802 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945774 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:13.945802 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.945786 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:13.947750 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.947732 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:13.947851 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.947759 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:13.948421 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.948403 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:13.948510 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.948437 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:13.948510 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.948452 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:13.975793 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.975778 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" node="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:13.977464 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.977449 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:13.978277 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:13.978260 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" node="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:13.978712 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:13.978696 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:14.076825 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.076805 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:14.078172 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.078157 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:14.078172 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.078171 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.078313 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:14.078188 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-60.us-west-1.compute.internal\": node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" Oct 31 14:45:14.087654 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.087629 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.088872 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.088855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71acc277a69aa4c1becf340e4de5eca3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"71acc277a69aa4c1becf340e4de5eca3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.088921 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.088881 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.088921 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.088899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.189125 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.189056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.189125 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.189093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.189311 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.189121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71acc277a69aa4c1becf340e4de5eca3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"71acc277a69aa4c1becf340e4de5eca3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.189311 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.189180 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71acc277a69aa4c1becf340e4de5eca3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"71acc277a69aa4c1becf340e4de5eca3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.189311 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.189188 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.189311 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.189192 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.194636 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.194616 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:14.195299 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.195288 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.280633 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.280597 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.281682 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.281668 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:14.300697 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.300681 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:14.623788 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.623713 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 14:45:14.624301 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.623851 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 14:45:14.624301 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.623903 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 14:45:14.624301 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.623919 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 14:45:14.624301 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:14.623918 2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-60.us-west-1.compute.internal.18e3a4786a2b7a1c\": read tcp 10.0.131.60:47340->13.59.157.100:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-60.us-west-1.compute.internal.18e3a4786a2b7a1c default 4918 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-60.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-60.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-60.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 14:45:13 +0000 UTC,LastTimestamp:2026-10-31 14:45:13.854655946 +0000 UTC m=+0.527339701,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-60.us-west-1.compute.internal,}" Oct 31 14:45:14.661677 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.661651 2570 apiserver.go:52] "Watching apiserver" Oct 31 14:45:14.687563 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.687531 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 14:45:14.901841 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.901773 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 14:40:13 +0000 UTC" deadline="2028-05-30 05:42:52.72062247 +0000 UTC" Oct 31 14:45:14.901841 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.901798 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13838h57m37.81882658s" Oct 31 14:45:14.937489 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.937444 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 14:45:14.938350 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.938323 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-l8qs8","openshift-multus/multus-additional-cni-plugins-n64fd","openshift-multus/network-metrics-daemon-6jlkx","openshift-network-diagnostics/network-check-target-jrfkr","openshift-network-operator/iptables-alerter-g7z6v","kube-system/global-pull-secret-syncer-h78q8","kube-system/konnectivity-agent-2bgb4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal","openshift-image-registry/node-ca-zgrl5","openshift-ovn-kubernetes/ovnkube-node-4knfg","kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k","openshift-cluster-node-tuning-operator/tuned-csvn8","openshift-dns/node-resolver-dfq2r"] Oct 31 14:45:14.941032 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.941007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:14.941121 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:14.941099 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:14.943086 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.943067 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.945185 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.945168 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:14.947193 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.947181 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:14.947257 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:14.947242 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:14.949210 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.949198 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:14.951261 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.951246 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:14.951321 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:14.951294 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:14.953347 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.953330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.955364 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.955349 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:14.957549 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.957530 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.959781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.959763 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.962281 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.962034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.964251 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.964235 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:14.989651 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.989632 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 14:45:14.995215 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995182 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-netns\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.995293 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-env-overrides\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.995293 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995256 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.995293 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-registration-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.995395 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-system-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.995395 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-k8s-cni-cncf-io\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.995395 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-multus-daemon-config\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.995521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995410 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-sys\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.995521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-systemd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.995521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995486 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cnibin\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.995521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995518 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:14.995706 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bafcc61-160a-4f2a-9081-29bd47254408-host\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:14.995706 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-netns\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.995706 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995604 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-multus-certs\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.995706 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-systemd\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.995706 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-script-lib\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.995706 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995686 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/959b8335-545a-4af4-a81b-8e4efee557e8-hosts-file\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-var-lib-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-node-log\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9tt\" (UniqueName: \"kubernetes.io/projected/0a562fc8-6e09-49d5-9dcf-3188700438eb-kube-api-access-rp9tt\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-agent-certs\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-kubelet-config\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-modprobe-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysconfig\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995863 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-var-lib-kubelet\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995885 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9tp\" (UniqueName: \"kubernetes.io/projected/58e34a13-1454-4b53-9667-23944e457c8d-kube-api-access-pg9tp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-slash\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.995917 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995918 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwjg\" (UniqueName: \"kubernetes.io/projected/7d91b481-174f-488b-97d8-d9a1c716a71d-kube-api-access-8qwjg\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-kubelet\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995965 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.995995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-config\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996044 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-lib-modules\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996082 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tdm\" (UniqueName: \"kubernetes.io/projected/959b8335-545a-4af4-a81b-8e4efee557e8-kube-api-access-d8tdm\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996112 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bafcc61-160a-4f2a-9081-29bd47254408-serviceca\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-socket-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-device-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-sys-fs\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-kubernetes\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-log-socket\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996238 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-netd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996252 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdlz\" (UniqueName: \"kubernetes.io/projected/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-kube-api-access-jgdlz\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.996328 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996311 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-bin\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996345 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1152975-ff11-49bf-b390-022434aae2a0-ovn-node-metrics-cert\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996384 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-cni-binary-copy\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-socket-dir-parent\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996449 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-multus\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996499 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d91b481-174f-488b-97d8-d9a1c716a71d-host-slash\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-etc-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996552 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttct\" (UniqueName: \"kubernetes.io/projected/f1152975-ff11-49bf-b390-022434aae2a0-kube-api-access-kttct\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996576 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cff\" (UniqueName: \"kubernetes.io/projected/40101b79-243a-4844-be24-6ecd1b07f67a-kube-api-access-c8cff\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996616 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-konnectivity-ca\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996639 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-cnibin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-conf-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996682 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996720 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b8335-545a-4af4-a81b-8e4efee557e8-tmp-dir\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-systemd-units\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.997058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996764 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-system-cni-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996789 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996831 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfktw\" (UniqueName: \"kubernetes.io/projected/0bafcc61-160a-4f2a-9081-29bd47254408-kube-api-access-jfktw\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996882 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996940 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d91b481-174f-488b-97d8-d9a1c716a71d-iptables-alerter-script\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-tmp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996973 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-ovn\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.996990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-kubelet\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997026 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-hostroot\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997046 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8bmx\" (UniqueName: \"kubernetes.io/projected/788d00c1-792d-47ea-bcd3-cfee09779724-kube-api-access-n8bmx\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-conf\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-run\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997112 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-host\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-dbus\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-bin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.997731 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997202 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997233 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997250 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-etc-kubernetes\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-etc-tuned\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997740 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997782 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997807 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.997989 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.998009 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.998057 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" Oct 31 14:45:14.998674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:14.998120 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" Oct 31 14:45:15.001861 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.001844 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" Oct 31 14:45:15.001955 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.001874 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:15.001955 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.001877 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:15.002020 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.001976 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 14:45:15.006460 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.006448 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 14:45:15.044866 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.044844 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 14:45:15.044942 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.044898 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pzcbk" Oct 31 14:45:15.097627 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-registration-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.097728 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-system-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097728 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097660 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-k8s-cni-cncf-io\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097728 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-multus-daemon-config\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097728 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097683 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-registration-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.097728 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097716 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-sys\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-systemd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cnibin\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097775 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-system-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bafcc61-160a-4f2a-9081-29bd47254408-host\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097782 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-sys\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097803 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cnibin\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-netns\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097813 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-k8s-cni-cncf-io\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097798 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-systemd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097846 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bafcc61-160a-4f2a-9081-29bd47254408-host\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-multus-certs\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097874 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-multus-certs\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-systemd\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097894 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-netns\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-script-lib\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097919 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 14:45:15.097961 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/959b8335-545a-4af4-a81b-8e4efee557e8-hosts-file\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097943 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-systemd\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-var-lib-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.097984 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-node-log\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098009 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/959b8335-545a-4af4-a81b-8e4efee557e8-hosts-file\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098014 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9tt\" (UniqueName: \"kubernetes.io/projected/0a562fc8-6e09-49d5-9dcf-3188700438eb-kube-api-access-rp9tt\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098020 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098025 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-var-lib-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098044 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098061 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-node-log\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-agent-certs\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098101 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098107 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098117 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-kubelet-config\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098143 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098146 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098117 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098150 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-modprobe-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098199 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-kubelet-config\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysconfig\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098258 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098260 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-modprobe-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.098850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098266 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098156 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098314 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098321 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098325 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-var-lib-kubelet\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098345 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysconfig\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098361 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9tp\" (UniqueName: \"kubernetes.io/projected/58e34a13-1454-4b53-9667-23944e457c8d-kube-api-access-pg9tp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-slash\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwjg\" (UniqueName: \"kubernetes.io/projected/7d91b481-174f-488b-97d8-d9a1c716a71d-kube-api-access-8qwjg\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098416 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098427 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-kubelet\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098332 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-var-lib-kubelet\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098459 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098463 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-slash\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098463 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098525 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-kubelet\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098529 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098496 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098528 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098552 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098560 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098492 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.100046 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098586 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098596 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-config\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098613 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-lib-modules\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098707 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tdm\" (UniqueName: \"kubernetes.io/projected/959b8335-545a-4af4-a81b-8e4efee557e8-kube-api-access-d8tdm\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bafcc61-160a-4f2a-9081-29bd47254408-serviceca\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098742 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-lib-modules\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098745 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-socket-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-device-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-sys-fs\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-kubernetes\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098873 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-log-socket\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-netd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdlz\" (UniqueName: \"kubernetes.io/projected/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-kube-api-access-jgdlz\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098926 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-kubernetes\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098940 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-device-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098850 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-socket-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098952 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-log-socket\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098898 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-sys-fs\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098998 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-netd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.098993 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-bin\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099036 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099040 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1152975-ff11-49bf-b390-022434aae2a0-ovn-node-metrics-cert\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099077 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-cni-binary-copy\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-socket-dir-parent\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099110 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bafcc61-160a-4f2a-9081-29bd47254408-serviceca\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099124 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-multus\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d91b481-174f-488b-97d8-d9a1c716a71d-host-slash\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099162 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-config\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099124 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-bin\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-etc-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.101904 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099203 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-etc-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttct\" (UniqueName: \"kubernetes.io/projected/f1152975-ff11-49bf-b390-022434aae2a0-kube-api-access-kttct\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099211 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d91b481-174f-488b-97d8-d9a1c716a71d-host-slash\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cff\" (UniqueName: \"kubernetes.io/projected/40101b79-243a-4844-be24-6ecd1b07f67a-kube-api-access-c8cff\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099243 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-multus\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099243 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-socket-dir-parent\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099255 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-konnectivity-ca\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-cnibin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-conf-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099313 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b8335-545a-4af4-a81b-8e4efee557e8-tmp-dir\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-systemd-units\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099372 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-cnibin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099392 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-systemd-units\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099487 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-conf-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099498 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099534 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-system-cni-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:15.102589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099612 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-system-cni-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099647 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-cni-binary-copy\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.099658 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099663 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b8335-545a-4af4-a81b-8e4efee557e8-tmp-dir\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099692 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfktw\" (UniqueName: \"kubernetes.io/projected/0bafcc61-160a-4f2a-9081-29bd47254408-kube-api-access-jfktw\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.099733 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:15.599715583 +0000 UTC m=+2.272399342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099865 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.099913 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d91b481-174f-488b-97d8-d9a1c716a71d-iptables-alerter-script\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-tmp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-ovn\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.099988 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.100016 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret podName:8eefaca9-ad88-44ac-be6b-363497696b72 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:15.600001044 +0000 UTC m=+2.272684789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret") pod "global-pull-secret-syncer-h78q8" (UID: "8eefaca9-ad88-44ac-be6b-363497696b72") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100035 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-ovn\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.103083 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-kubelet\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100078 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-hostroot\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100095 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-konnectivity-ca\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8bmx\" (UniqueName: \"kubernetes.io/projected/788d00c1-792d-47ea-bcd3-cfee09779724-kube-api-access-n8bmx\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-conf\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100146 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-kubelet\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100151 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-hostroot\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-run\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100189 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-run\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-host\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-dbus\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100234 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-host\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-bin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100363 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-conf\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100368 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-etc-kubernetes\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.103584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100407 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-etc-tuned\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100424 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-dbus\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100435 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-netns\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-env-overrides\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100512 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100892 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100426 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d91b481-174f-488b-97d8-d9a1c716a71d-iptables-alerter-script\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100960 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-netns\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100378 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-bin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.101107 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.101140 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.100463 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-etc-kubernetes\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.101138 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.101840 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-env-overrides\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.102387 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-tmp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.102882 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-etc-tuned\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.102990 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1152975-ff11-49bf-b390-022434aae2a0-ovn-node-metrics-cert\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.104058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.103071 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-agent-certs\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:15.109017 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.108996 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-multus-daemon-config\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.141146 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.141122 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 14:45:15.148589 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.148569 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-script-lib\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.154190 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.154149 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:15.154190 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.154169 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:15.154190 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.154179 2570 projected.go:196] Error preparing data for projected volume kube-api-access-75kjc for pod openshift-network-diagnostics/network-check-target-jrfkr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:15.154367 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.154241 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc podName:07979459-61ec-49fc-9696-2c985d7ad790 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:15.654228166 +0000 UTC m=+2.326911927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-75kjc" (UniqueName: "kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc") pod "network-check-target-jrfkr" (UID: "07979459-61ec-49fc-9696-2c985d7ad790") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:15.156414 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.156343 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwjg\" (UniqueName: \"kubernetes.io/projected/7d91b481-174f-488b-97d8-d9a1c716a71d-kube-api-access-8qwjg\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:15.157058 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.157035 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9tt\" (UniqueName: \"kubernetes.io/projected/0a562fc8-6e09-49d5-9dcf-3188700438eb-kube-api-access-rp9tt\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.158445 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.158422 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfktw\" (UniqueName: \"kubernetes.io/projected/0bafcc61-160a-4f2a-9081-29bd47254408-kube-api-access-jfktw\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:15.159076 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.159036 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdlz\" (UniqueName: \"kubernetes.io/projected/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-kube-api-access-jgdlz\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.159217 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.159196 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cff\" (UniqueName: \"kubernetes.io/projected/40101b79-243a-4844-be24-6ecd1b07f67a-kube-api-access-c8cff\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:15.159329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.159308 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9tp\" (UniqueName: \"kubernetes.io/projected/58e34a13-1454-4b53-9667-23944e457c8d-kube-api-access-pg9tp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.159386 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.159353 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8bmx\" (UniqueName: \"kubernetes.io/projected/788d00c1-792d-47ea-bcd3-cfee09779724-kube-api-access-n8bmx\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.160864 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.160842 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tdm\" (UniqueName: \"kubernetes.io/projected/959b8335-545a-4af4-a81b-8e4efee557e8-kube-api-access-d8tdm\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:15.161057 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.161040 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttct\" (UniqueName: \"kubernetes.io/projected/f1152975-ff11-49bf-b390-022434aae2a0-kube-api-access-kttct\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.272372 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.272342 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:15.285243 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.285214 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:15.290834 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.290817 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:15.308310 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.308289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8qs8" Oct 31 14:45:15.331915 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.331897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:15.339538 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.339510 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788d00c1_792d_47ea_bcd3_cfee09779724.slice/crio-c7b29bf60ed9315b159e46541de7ec1007b913a2aaceea4c97757f98c4f09569 WatchSource:0}: Error finding container c7b29bf60ed9315b159e46541de7ec1007b913a2aaceea4c97757f98c4f09569: Status 404 returned error can't find the container with id c7b29bf60ed9315b159e46541de7ec1007b913a2aaceea4c97757f98c4f09569 Oct 31 14:45:15.340223 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.340203 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9c00f3a8edeac7e2ac58fb07ea319e.slice/crio-394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1 WatchSource:0}: Error finding container 394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1: Status 404 returned error can't find the container with id 394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1 Oct 31 14:45:15.346365 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.346349 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:45:15.352741 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.352718 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bafcc61_160a_4f2a_9081_29bd47254408.slice/crio-8c7ab459cc5dcc8cc6006449b48842a0fd69b5f2d5bed6e112431d56446f0049 WatchSource:0}: Error finding container 8c7ab459cc5dcc8cc6006449b48842a0fd69b5f2d5bed6e112431d56446f0049: Status 404 returned error can't find the container with id 8c7ab459cc5dcc8cc6006449b48842a0fd69b5f2d5bed6e112431d56446f0049 Oct 31 14:45:15.353913 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.353388 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71acc277a69aa4c1becf340e4de5eca3.slice/crio-aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382 WatchSource:0}: Error finding container aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382: Status 404 returned error can't find the container with id aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382 Oct 31 14:45:15.367560 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.367542 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:15.376002 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.375982 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1152975_ff11_49bf_b390_022434aae2a0.slice/crio-f83dc6a977e4125f7941bbaf63776685e36b83f5cd9ea128b9020f92b9632244 WatchSource:0}: Error finding container f83dc6a977e4125f7941bbaf63776685e36b83f5cd9ea128b9020f92b9632244: Status 404 returned error can't find the container with id f83dc6a977e4125f7941bbaf63776685e36b83f5cd9ea128b9020f92b9632244 Oct 31 14:45:15.381737 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.381718 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:15.385441 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.385427 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:15.390395 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.390379 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefbecfb_5e0b_4d8b_8e13_54c7394ba5bc.slice/crio-fd4a293ac95cf4a9bbabe7be1f8ecf27627505e72ddfbff13a8eb635fe7ffcde WatchSource:0}: Error finding container fd4a293ac95cf4a9bbabe7be1f8ecf27627505e72ddfbff13a8eb635fe7ffcde: Status 404 returned error can't find the container with id fd4a293ac95cf4a9bbabe7be1f8ecf27627505e72ddfbff13a8eb635fe7ffcde Oct 31 14:45:15.393460 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.393444 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:15.394112 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.394087 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e34a13_1454_4b53_9667_23944e457c8d.slice/crio-f8f779a44767b2961e85fe4c26c91c87a0c37642e45773d872fe7226c795cf36 WatchSource:0}: Error finding container f8f779a44767b2961e85fe4c26c91c87a0c37642e45773d872fe7226c795cf36: Status 404 returned error can't find the container with id f8f779a44767b2961e85fe4c26c91c87a0c37642e45773d872fe7226c795cf36 Oct 31 14:45:15.402123 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:15.402104 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959b8335_545a_4af4_a81b_8e4efee557e8.slice/crio-77c985bb831b42140dac8051b5e9e9a3af7a488317b0461ab3e4045b6c27b5d9 WatchSource:0}: Error finding container 77c985bb831b42140dac8051b5e9e9a3af7a488317b0461ab3e4045b6c27b5d9: Status 404 returned error can't find the container with id 77c985bb831b42140dac8051b5e9e9a3af7a488317b0461ab3e4045b6c27b5d9 Oct 31 14:45:15.604376 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.604344 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:15.604548 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.604384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:15.604548 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.604505 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:15.604548 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.604507 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:15.604652 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.604564 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret podName:8eefaca9-ad88-44ac-be6b-363497696b72 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:16.604552004 +0000 UTC m=+3.277235745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret") pod "global-pull-secret-syncer-h78q8" (UID: "8eefaca9-ad88-44ac-be6b-363497696b72") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:15.604652 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.604578 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:16.604572014 +0000 UTC m=+3.277255754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:15.687819 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.687745 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:15.705575 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.705546 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:15.705714 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.705698 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:15.705753 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.705721 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:15.705753 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.705732 2570 projected.go:196] Error preparing data for projected volume kube-api-access-75kjc for pod openshift-network-diagnostics/network-check-target-jrfkr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:15.705812 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:15.705776 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc podName:07979459-61ec-49fc-9696-2c985d7ad790 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:16.705763656 +0000 UTC m=+3.378447397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-75kjc" (UniqueName: "kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc") pod "network-check-target-jrfkr" (UID: "07979459-61ec-49fc-9696-2c985d7ad790") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:15.814845 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.814818 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:15.846172 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.846130 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"f83dc6a977e4125f7941bbaf63776685e36b83f5cd9ea128b9020f92b9632244"} Oct 31 14:45:15.847143 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.847123 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"71acc277a69aa4c1becf340e4de5eca3","Type":"ContainerStarted","Data":"aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382"} Oct 31 14:45:15.848247 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.848219 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zgrl5" event={"ID":"0bafcc61-160a-4f2a-9081-29bd47254408","Type":"ContainerStarted","Data":"8c7ab459cc5dcc8cc6006449b48842a0fd69b5f2d5bed6e112431d56446f0049"} Oct 31 14:45:15.849335 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.849309 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"3e9c00f3a8edeac7e2ac58fb07ea319e","Type":"ContainerStarted","Data":"394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1"} Oct 31 14:45:15.850339 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.850322 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8qs8" event={"ID":"788d00c1-792d-47ea-bcd3-cfee09779724","Type":"ContainerStarted","Data":"c7b29bf60ed9315b159e46541de7ec1007b913a2aaceea4c97757f98c4f09569"} Oct 31 14:45:15.851361 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.851342 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dfq2r" event={"ID":"959b8335-545a-4af4-a81b-8e4efee557e8","Type":"ContainerStarted","Data":"77c985bb831b42140dac8051b5e9e9a3af7a488317b0461ab3e4045b6c27b5d9"} Oct 31 14:45:15.852373 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.852347 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-csvn8" event={"ID":"58e34a13-1454-4b53-9667-23944e457c8d","Type":"ContainerStarted","Data":"f8f779a44767b2961e85fe4c26c91c87a0c37642e45773d872fe7226c795cf36"} Oct 31 14:45:15.853318 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:15.853296 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerStarted","Data":"fd4a293ac95cf4a9bbabe7be1f8ecf27627505e72ddfbff13a8eb635fe7ffcde"} Oct 31 14:45:16.047466 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.047369 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:14 +0000 UTC" deadline="2028-05-06 06:13:02.991672699 +0000 UTC" Oct 31 14:45:16.047466 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.047423 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13263h27m46.944253553s" Oct 31 14:45:16.077933 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.077903 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:16.084246 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:16.084217 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a562fc8_6e09_49d5_9dcf_3188700438eb.slice/crio-4ec2aec1f1e48c4041a1650aa97ba69790060028d15d401763b71e00acf483c4 WatchSource:0}: Error finding container 4ec2aec1f1e48c4041a1650aa97ba69790060028d15d401763b71e00acf483c4: Status 404 returned error can't find the container with id 4ec2aec1f1e48c4041a1650aa97ba69790060028d15d401763b71e00acf483c4 Oct 31 14:45:16.084886 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:16.084865 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9b42eb_56a8_4c76_9fbe_4d2c21d2e9f4.slice/crio-1c2419a8387d77a20bfd3f9be4e8c2baddb018d8745b9eb0608b86b7fd8537b0 WatchSource:0}: Error finding container 1c2419a8387d77a20bfd3f9be4e8c2baddb018d8745b9eb0608b86b7fd8537b0: Status 404 returned error can't find the container with id 1c2419a8387d77a20bfd3f9be4e8c2baddb018d8745b9eb0608b86b7fd8537b0 Oct 31 14:45:16.102375 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:16.102356 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d91b481_174f_488b_97d8_d9a1c716a71d.slice/crio-fbe864b6740ca9d006f3fb0168700f5bd1d64a30329ae07ac4975440678b1a87 WatchSource:0}: Error finding container fbe864b6740ca9d006f3fb0168700f5bd1d64a30329ae07ac4975440678b1a87: Status 404 returned error can't find the container with id fbe864b6740ca9d006f3fb0168700f5bd1d64a30329ae07ac4975440678b1a87 Oct 31 14:45:16.612295 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.612254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:16.612521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.612303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:16.612521 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.612444 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:16.612521 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.612518 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret podName:8eefaca9-ad88-44ac-be6b-363497696b72 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:18.612500272 +0000 UTC m=+5.285184018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret") pod "global-pull-secret-syncer-h78q8" (UID: "8eefaca9-ad88-44ac-be6b-363497696b72") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:16.612720 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.612597 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:16.612720 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.612650 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:18.612637179 +0000 UTC m=+5.285320935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:16.713204 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.713168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:16.713872 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.713340 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:16.713872 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.713367 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:16.713872 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.713380 2570 projected.go:196] Error preparing data for projected volume kube-api-access-75kjc for pod openshift-network-diagnostics/network-check-target-jrfkr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:16.713872 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.713437 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc podName:07979459-61ec-49fc-9696-2c985d7ad790 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:18.713418334 +0000 UTC m=+5.386102079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-75kjc" (UniqueName: "kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc") pod "network-check-target-jrfkr" (UID: "07979459-61ec-49fc-9696-2c985d7ad790") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:16.842060 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.842026 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:16.842268 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.842154 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:16.842268 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.842244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:16.842516 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.842329 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:16.842516 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.842386 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:16.842516 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:16.842494 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:16.873858 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.873783 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7z6v" event={"ID":"7d91b481-174f-488b-97d8-d9a1c716a71d","Type":"ContainerStarted","Data":"fbe864b6740ca9d006f3fb0168700f5bd1d64a30329ae07ac4975440678b1a87"} Oct 31 14:45:16.900102 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.900068 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2bgb4" event={"ID":"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4","Type":"ContainerStarted","Data":"1c2419a8387d77a20bfd3f9be4e8c2baddb018d8745b9eb0608b86b7fd8537b0"} Oct 31 14:45:16.916231 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:16.916176 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" event={"ID":"0a562fc8-6e09-49d5-9dcf-3188700438eb","Type":"ContainerStarted","Data":"4ec2aec1f1e48c4041a1650aa97ba69790060028d15d401763b71e00acf483c4"} Oct 31 14:45:17.047663 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:17.047574 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:14 +0000 UTC" deadline="2028-05-27 18:36:12.624576582 +0000 UTC" Oct 31 14:45:17.047663 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:17.047606 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13779h50m55.576974356s" Oct 31 14:45:18.629221 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:18.629184 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:18.629799 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:18.629233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:18.629799 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.629387 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:18.629799 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.629459 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret podName:8eefaca9-ad88-44ac-be6b-363497696b72 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:22.629441729 +0000 UTC m=+9.302125486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret") pod "global-pull-secret-syncer-h78q8" (UID: "8eefaca9-ad88-44ac-be6b-363497696b72") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:18.629980 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.629801 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:18.629980 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.629853 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:22.629836477 +0000 UTC m=+9.302520221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:18.730734 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:18.730693 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:18.730885 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.730874 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:18.730922 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.730896 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:18.730922 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.730910 2570 projected.go:196] Error preparing data for projected volume kube-api-access-75kjc for pod openshift-network-diagnostics/network-check-target-jrfkr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:18.730989 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.730968 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc podName:07979459-61ec-49fc-9696-2c985d7ad790 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:22.73095177 +0000 UTC m=+9.403635516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-75kjc" (UniqueName: "kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc") pod "network-check-target-jrfkr" (UID: "07979459-61ec-49fc-9696-2c985d7ad790") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:18.842367 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:18.842298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:18.842530 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.842412 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:18.842612 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:18.842565 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:18.842670 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.842651 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:18.842722 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:18.842707 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:18.842850 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:18.842789 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:19.937310 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:19.936358 2570 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="5633decd426dafc0266c4b44f0f4dd0747ca56df60a2f64b76d9a79eeac22e52" exitCode=0 Oct 31 14:45:19.937310 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:19.936615 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerDied","Data":"5633decd426dafc0266c4b44f0f4dd0747ca56df60a2f64b76d9a79eeac22e52"} Oct 31 14:45:20.842401 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:20.842371 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:20.842584 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:20.842525 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:20.842632 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:20.842601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:20.842713 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:20.842692 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:20.842791 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:20.842766 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:20.842859 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:20.842840 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:20.948230 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:20.948193 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"71acc277a69aa4c1becf340e4de5eca3","Type":"ContainerStarted","Data":"6c8c3ba70f4b358924bef6cda2d71ca15ddad854159515095a7dd315ec9000e9"} Oct 31 14:45:20.955399 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:20.955371 2570 generic.go:356] "Generic (PLEG): container finished" podID="3e9c00f3a8edeac7e2ac58fb07ea319e" containerID="aa8b5a6b51cc75920d8f810c1c3b609c377164ac611b0b647b9c98cee560d3f0" exitCode=0 Oct 31 14:45:20.956164 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:20.955502 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"3e9c00f3a8edeac7e2ac58fb07ea319e","Type":"ContainerDied","Data":"aa8b5a6b51cc75920d8f810c1c3b609c377164ac611b0b647b9c98cee560d3f0"} Oct 31 14:45:21.062940 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:21.062874 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" podStartSLOduration=7.06285585 podStartE2EDuration="7.06285585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:21.062164064 +0000 UTC m=+7.734847828" watchObservedRunningTime="2026-10-31 14:45:21.06285585 +0000 UTC m=+7.735539612" Oct 31 14:45:21.961845 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:21.961806 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"3e9c00f3a8edeac7e2ac58fb07ea319e","Type":"ContainerStarted","Data":"1a6bb78cdb206eb4f92bf29f97e5f6878b3ec98d806d0764cdf63adc172f1156"} Oct 31 14:45:21.965380 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:21.965348 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zgrl5" event={"ID":"0bafcc61-160a-4f2a-9081-29bd47254408","Type":"ContainerStarted","Data":"7b3c32f1473cc16047dcca4d3b199eda3e55b82a41da21fd94696574fe1dd39a"} Oct 31 14:45:21.969613 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:21.969588 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2bgb4" event={"ID":"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4","Type":"ContainerStarted","Data":"43eed77fc3e925f453b7ceb72ff63447d11cf7206f4bfdc2d439419df6c9d7f2"} Oct 31 14:45:22.085233 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.084720 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" podStartSLOduration=8.084702611 podStartE2EDuration="8.084702611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:22.083848151 +0000 UTC m=+8.756531915" watchObservedRunningTime="2026-10-31 14:45:22.084702611 +0000 UTC m=+8.757386375" Oct 31 14:45:22.171602 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.171528 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:22.313567 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.313450 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2bgb4" podStartSLOduration=3.381899667 podStartE2EDuration="8.313431336s" totalImagesPullingTime="4.931531669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:22.312897858 +0000 UTC m=+8.985581625" watchObservedRunningTime="2026-10-31 14:45:22.313431336 +0000 UTC m=+8.986115110" Oct 31 14:45:22.313735 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.313615 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zgrl5" podStartSLOduration=2.576014674 podStartE2EDuration="8.313607925s" totalImagesPullingTime="5.737593251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:22.199464076 +0000 UTC m=+8.872148145" watchObservedRunningTime="2026-10-31 14:45:22.313607925 +0000 UTC m=+8.986291689" Oct 31 14:45:22.662818 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.662086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:22.662818 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.662191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:22.662818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.662312 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:22.662818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.662374 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:30.662354611 +0000 UTC m=+17.335038376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:22.662818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.662438 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:22.662818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.662488 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret podName:8eefaca9-ad88-44ac-be6b-363497696b72 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:30.662458346 +0000 UTC m=+17.335142091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret") pod "global-pull-secret-syncer-h78q8" (UID: "8eefaca9-ad88-44ac-be6b-363497696b72") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:22.762781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.762724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:22.763188 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.762942 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:22.763188 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.762963 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:22.763188 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.762976 2570 projected.go:196] Error preparing data for projected volume kube-api-access-75kjc for pod openshift-network-diagnostics/network-check-target-jrfkr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:22.763188 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.763033 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc podName:07979459-61ec-49fc-9696-2c985d7ad790 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:30.763015393 +0000 UTC m=+17.435699137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-75kjc" (UniqueName: "kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc") pod "network-check-target-jrfkr" (UID: "07979459-61ec-49fc-9696-2c985d7ad790") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:22.842631 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.842508 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:22.842631 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.842529 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:22.842806 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.842641 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:22.842806 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.842721 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:22.842806 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.842773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:22.842896 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:22.842849 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:22.974281 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:22.974245 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" event={"ID":"0a562fc8-6e09-49d5-9dcf-3188700438eb","Type":"ContainerStarted","Data":"5bd9698927da32a94ab6472157a8e2fe5e48ea3dbbcc1ea98fd11456d02fc60b"} Oct 31 14:45:24.747840 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.747811 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 14:45:24.758764 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.758674 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T14:45:24.747834875Z","UUID":"865b5e42-c49b-4cd6-9a70-33e7af5e9db7","Handler":null,"Name":"","Endpoint":""} Oct 31 14:45:24.760585 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.760223 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 14:45:24.760585 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.760251 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 14:45:24.841795 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.841621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:24.841795 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.841738 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:24.841795 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.841767 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:24.842004 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:24.841743 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:24.842004 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:24.841869 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:24.842004 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:24.841953 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:24.981248 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.981211 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8qs8" event={"ID":"788d00c1-792d-47ea-bcd3-cfee09779724","Type":"ContainerStarted","Data":"f8cf9d1cb78bffd5b4155150dcd12942b4d8b0bba28b0ae43fc77e8a210a0365"} Oct 31 14:45:24.983702 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.983634 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" event={"ID":"0a562fc8-6e09-49d5-9dcf-3188700438eb","Type":"ContainerStarted","Data":"1db8039e8070a75b1f58d52bb9674f23bdb52999815d58000381dee2e15a4e57"} Oct 31 14:45:24.985790 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.985519 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dfq2r" event={"ID":"959b8335-545a-4af4-a81b-8e4efee557e8","Type":"ContainerStarted","Data":"df4a77faa24a77cadb703b3900f416cd4a9f45f3f4c7c1ebd9ddda7bee4e247e"} Oct 31 14:45:24.988152 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:24.988125 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerStarted","Data":"914b9da3849499dd2fde2790ea44705408493239bd957330142f45f4563d581c"} Oct 31 14:45:25.099791 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:25.099729 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-l8qs8" podStartSLOduration=2.5059077739999998 podStartE2EDuration="11.099712941s" totalImagesPullingTime="8.593805167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:25.099617435 +0000 UTC m=+11.772301200" watchObservedRunningTime="2026-10-31 14:45:25.099712941 +0000 UTC m=+11.772396706" Oct 31 14:45:25.944321 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:25.944229 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:25.945025 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:25.945007 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:25.992641 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:25.992609 2570 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="914b9da3849499dd2fde2790ea44705408493239bd957330142f45f4563d581c" exitCode=0 Oct 31 14:45:25.992811 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:25.992738 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerDied","Data":"914b9da3849499dd2fde2790ea44705408493239bd957330142f45f4563d581c"} Oct 31 14:45:25.993780 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:25.993543 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:26.068211 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:26.068151 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dfq2r" podStartSLOduration=3.377587209 podStartE2EDuration="12.068136661s" totalImagesPullingTime="8.690549452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:25.339729376 +0000 UTC m=+12.012413138" watchObservedRunningTime="2026-10-31 14:45:26.068136661 +0000 UTC m=+12.740820425" Oct 31 14:45:26.841732 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:26.841570 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:26.841899 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:26.841621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:26.841899 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:26.841797 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:26.841899 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:26.841621 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:26.841989 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:26.841866 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:26.841989 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:26.841963 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:26.996026 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:26.995993 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g7z6v" event={"ID":"7d91b481-174f-488b-97d8-d9a1c716a71d","Type":"ContainerStarted","Data":"e0af00975765871b3e7b20c2fde96e9c0fcb4d280133061a0e9ffd4f0a7b2ed0"} Oct 31 14:45:26.997749 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:26.997719 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-csvn8" event={"ID":"58e34a13-1454-4b53-9667-23944e457c8d","Type":"ContainerStarted","Data":"0177798f1828ca1236c7ba9e6388de4f771c0278125cf9449733db644d8fb812"} Oct 31 14:45:27.000575 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:27.000535 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" event={"ID":"0a562fc8-6e09-49d5-9dcf-3188700438eb","Type":"ContainerStarted","Data":"2aa3bb8b628f7c1f4d555f88be1adbea8201fd80179124e371fa2d1764841531"} Oct 31 14:45:27.111246 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:27.111139 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g7z6v" podStartSLOduration=4.546940756 podStartE2EDuration="13.111119843s" totalImagesPullingTime="8.564179087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:27.110832295 +0000 UTC m=+13.783516057" watchObservedRunningTime="2026-10-31 14:45:27.111119843 +0000 UTC m=+13.783803605" Oct 31 14:45:27.228089 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:27.228029 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-csvn8" podStartSLOduration=2.473549044 podStartE2EDuration="13.228012351s" totalImagesPullingTime="10.754463307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:27.227292448 +0000 UTC m=+13.899976213" watchObservedRunningTime="2026-10-31 14:45:27.228012351 +0000 UTC m=+13.900696115" Oct 31 14:45:27.347462 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:27.347399 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" podStartSLOduration=4.541753224 podStartE2EDuration="14.347380115s" totalImagesPullingTime="9.805626891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:27.346579583 +0000 UTC m=+14.019263347" watchObservedRunningTime="2026-10-31 14:45:27.347380115 +0000 UTC m=+14.020063881" Oct 31 14:45:28.005019 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:28.004983 2570 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="933182eebc1e2ef656158d3169faa88c62fa64b0515449b17d39c4506ffa195b" exitCode=0 Oct 31 14:45:28.005693 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:28.005090 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerDied","Data":"933182eebc1e2ef656158d3169faa88c62fa64b0515449b17d39c4506ffa195b"} Oct 31 14:45:28.841677 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:28.841645 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:28.841843 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:28.841693 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:28.841843 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:28.841800 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:28.841955 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:28.841877 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:28.841955 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:28.841906 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:28.842064 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:28.842015 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:30.011086 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.011056 2570 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="56eb5cd9c8408fd838cabca31c8a9d22056e1d933b80291a21a49c6e60e8e19a" exitCode=0 Oct 31 14:45:30.011086 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.011093 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerDied","Data":"56eb5cd9c8408fd838cabca31c8a9d22056e1d933b80291a21a49c6e60e8e19a"} Oct 31 14:45:30.722911 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.722719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:30.723005 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.722938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:30.723005 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.722855 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:30.723120 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.723033 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:46.723013645 +0000 UTC m=+33.395697390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:30.723120 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.723058 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:30.723120 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.723106 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret podName:8eefaca9-ad88-44ac-be6b-363497696b72 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:46.72309072 +0000 UTC m=+33.395774480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret") pod "global-pull-secret-syncer-h78q8" (UID: "8eefaca9-ad88-44ac-be6b-363497696b72") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:30.823254 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.823227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:30.823379 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.823364 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:30.823421 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.823382 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:30.823421 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.823391 2570 projected.go:196] Error preparing data for projected volume kube-api-access-75kjc for pod openshift-network-diagnostics/network-check-target-jrfkr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:30.823497 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.823431 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc podName:07979459-61ec-49fc-9696-2c985d7ad790 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:46.823419117 +0000 UTC m=+33.496102877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-75kjc" (UniqueName: "kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc") pod "network-check-target-jrfkr" (UID: "07979459-61ec-49fc-9696-2c985d7ad790") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:30.842343 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.842320 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:30.842447 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.842392 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:30.842447 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:30.842417 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:30.842538 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.842505 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:30.842579 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.842552 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:30.842669 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:30.842648 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:31.017374 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017301 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:45:31.017907 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017722 2570 generic.go:356] "Generic (PLEG): container finished" podID="f1152975-ff11-49bf-b390-022434aae2a0" containerID="f531565e6ec6483ac0f8cb529d19764638087e6a422d055be1187f2a631bfd5e" exitCode=1 Oct 31 14:45:31.017907 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017777 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"da7d4a6efd4dd084c20f44a451f448feb030d609462982dca8d33fb8e6acb41d"} Oct 31 14:45:31.017907 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017853 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"36c9ff8beb4b713b0671c7122529ebb44c85f48b7dcffd1fbb439171d7562974"} Oct 31 14:45:31.017907 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017885 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"289a12e3f26f05182dcf169d61e953d45fa36635155c7ad88d82a6881619de24"} Oct 31 14:45:31.018063 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017915 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"9569758b4e5053e421f5ca893c8e059e2509d9857afd2b262f1b4a9c1e1996e9"} Oct 31 14:45:31.018063 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017950 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerDied","Data":"f531565e6ec6483ac0f8cb529d19764638087e6a422d055be1187f2a631bfd5e"} Oct 31 14:45:31.018063 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:31.017992 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"40726463718d435131ba0bcfd8e043345360549acbb4c0038498718386e2bfce"} Oct 31 14:45:32.022617 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:32.022428 2570 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="fb11d93c1a46bba05e02eb362ec047fcc1acd18a45df7d7b6559ba6af3eb8666" exitCode=0 Oct 31 14:45:32.022617 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:32.022499 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerDied","Data":"fb11d93c1a46bba05e02eb362ec047fcc1acd18a45df7d7b6559ba6af3eb8666"} Oct 31 14:45:32.842634 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:32.842602 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:32.842821 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:32.842692 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:32.842821 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:32.842717 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:32.842821 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:32.842787 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:32.842953 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:32.842844 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:32.843001 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:32.842947 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:33.028071 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:33.028039 2570 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="de8df5b72f058179ad9fbb3018757db39dbf7ccc8ef20d28c6e9f7dad5bb69f4" exitCode=0 Oct 31 14:45:33.028564 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:33.028119 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerDied","Data":"de8df5b72f058179ad9fbb3018757db39dbf7ccc8ef20d28c6e9f7dad5bb69f4"} Oct 31 14:45:33.032441 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:33.032418 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:45:33.032925 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:33.032803 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"21f35453285102e6713ee1054486a6009c2d6a3b6206998b53b85b415e7ac568"} Oct 31 14:45:34.038135 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:34.038104 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n64fd" event={"ID":"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc","Type":"ContainerStarted","Data":"99b93105fffddc20271d335c503ef923084af1c5ad000d596b178f017ea6da70"} Oct 31 14:45:34.261263 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:34.260110 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n64fd" podStartSLOduration=7.695332218 podStartE2EDuration="20.260091998s" totalImagesPullingTime="12.56475978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:34.258947486 +0000 UTC m=+20.931631249" watchObservedRunningTime="2026-10-31 14:45:34.260091998 +0000 UTC m=+20.932775762" Oct 31 14:45:34.841762 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:34.841729 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:34.841943 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:34.841774 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:34.841943 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:34.841784 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:34.841943 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:34.841859 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:34.841943 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:34.841924 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:34.842105 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:34.842005 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:35.044916 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:35.044893 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:45:35.045507 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:35.045392 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"cad239d093361cf47923bd18d03511927861ec99df8aa6b58ed66f4dd2ffefb6"} Oct 31 14:45:35.046079 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:35.046064 2570 scope.go:122] "RemoveContainer" containerID="f531565e6ec6483ac0f8cb529d19764638087e6a422d055be1187f2a631bfd5e" Oct 31 14:45:36.050948 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.050775 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:45:36.051497 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.051283 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" event={"ID":"f1152975-ff11-49bf-b390-022434aae2a0","Type":"ContainerStarted","Data":"3f2d33821faa07f5c665240a5fc940d664812cc9492b97aa2fe16817520c47c4"} Oct 31 14:45:36.051587 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.051553 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:36.051644 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.051607 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:36.051714 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.051650 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:36.067422 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.067402 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:36.067760 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.067745 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:36.494006 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.493912 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" podStartSLOduration=7.502949706 podStartE2EDuration="22.493899667s" totalImagesPullingTime="14.990949961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:36.345504689 +0000 UTC m=+23.018188452" watchObservedRunningTime="2026-10-31 14:45:36.493899667 +0000 UTC m=+23.166583441" Oct 31 14:45:36.842408 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.842326 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:36.842600 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.842340 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:36.842600 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:36.842454 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:36.842600 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:36.842579 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:36.842759 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:36.842588 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:36.842759 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:36.842649 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:37.707598 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:37.707565 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jrfkr"] Oct 31 14:45:37.708008 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:37.707677 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:37.708008 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:37.707760 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:37.711251 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:37.711226 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h78q8"] Oct 31 14:45:37.711363 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:37.711298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:37.711399 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:37.711376 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:37.713626 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:37.713603 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jlkx"] Oct 31 14:45:37.713729 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:37.713683 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:37.713771 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:37.713760 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:38.841728 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:38.841697 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:38.842210 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:38.841751 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:38.842210 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:38.841754 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:38.842210 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:38.841853 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:38.842210 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:38.841909 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:38.842210 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:38.841957 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:40.842125 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:40.842095 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:40.842125 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:40.842125 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:40.842624 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:40.842202 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:40.842624 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:40.842249 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:40.842624 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:40.842317 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:40.842624 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:40.842407 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:42.842521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:42.842492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:42.842926 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:42.842492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:42.842926 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:42.842600 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jlkx" podUID="40101b79-243a-4844-be24-6ecd1b07f67a" Oct 31 14:45:42.842926 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:42.842492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:42.842926 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:42.842653 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h78q8" podUID="8eefaca9-ad88-44ac-be6b-363497696b72" Oct 31 14:45:42.842926 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:42.842745 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jrfkr" podUID="07979459-61ec-49fc-9696-2c985d7ad790" Oct 31 14:45:43.085086 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.085056 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeReady" Oct 31 14:45:43.085254 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.085244 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 14:45:43.198732 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.198705 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b4frh"] Oct 31 14:45:43.201752 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.201736 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5p7xg"] Oct 31 14:45:43.201861 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.201844 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b4frh" Oct 31 14:45:43.204854 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.204830 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gq2qd"] Oct 31 14:45:43.204938 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.204920 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" Oct 31 14:45:43.207897 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.207878 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m"] Oct 31 14:45:43.207994 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.207957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" Oct 31 14:45:43.210858 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.210839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.242667 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.242643 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-n4mj8"] Oct 31 14:45:43.245312 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.245296 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.249292 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.249273 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk"] Oct 31 14:45:43.251998 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.251980 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj"] Oct 31 14:45:43.252122 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.252104 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.254867 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.254850 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl"] Oct 31 14:45:43.254968 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.254955 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" Oct 31 14:45:43.258233 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.258214 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-699cdb64d9-r57rx"] Oct 31 14:45:43.258338 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.258319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" Oct 31 14:45:43.260827 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.260811 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-678477bddd-n4c79"] Oct 31 14:45:43.260947 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.260930 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.263659 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.263643 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gdnkp"] Oct 31 14:45:43.263772 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.263757 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.266433 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.266417 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7fv54"] Oct 31 14:45:43.266574 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.266559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.269230 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.269213 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:43.293672 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.293650 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ddb7677d5-drb9t"] Oct 31 14:45:43.297624 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.297600 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f"] Oct 31 14:45:43.298431 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.298385 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.299905 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.299888 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.300009 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.299938 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.300101 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.300084 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 14:45:43.300645 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.300624 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2"] Oct 31 14:45:43.301461 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.301445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.301872 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.301855 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 14:45:43.302071 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.302054 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 14:45:43.302235 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.302217 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whbtk\"" Oct 31 14:45:43.302298 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.302218 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gm7jd\"" Oct 31 14:45:43.305571 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.305550 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sptz6"] Oct 31 14:45:43.308227 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.308205 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.311943 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.309667 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mg2kx\"" Oct 31 14:45:43.314565 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.313374 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vh7rz"] Oct 31 14:45:43.314565 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.313601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.318696 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318675 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5p7xg"] Oct 31 14:45:43.318781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318714 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b4frh"] Oct 31 14:45:43.318781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318735 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gq2qd"] Oct 31 14:45:43.318781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318749 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m"] Oct 31 14:45:43.318781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318762 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk"] Oct 31 14:45:43.318781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318775 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-n4mj8"] Oct 31 14:45:43.318781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318777 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.318781 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318786 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj"] Oct 31 14:45:43.319097 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.318799 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl"] Oct 31 14:45:43.320394 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320371 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv59l\" (UniqueName: \"kubernetes.io/projected/f3b0ed9d-d90a-497a-8863-7605a3c89c75-kube-api-access-gv59l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.320511 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.320511 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320441 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.320620 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320526 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477-kube-api-access-bqplf\") pod \"volume-data-source-validator-58c9f5f46d-4lmnj\" (UID: \"a8319738-e6b9-47aa-9ceb-5f0bf4d5f477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" Oct 31 14:45:43.320656 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320643 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfxw\" (UniqueName: \"kubernetes.io/projected/14236efa-49b0-452b-8136-157cdb6390cf-kube-api-access-rkfxw\") pod \"kas-connection-checker-685895c7c9-5p7xg\" (UID: \"14236efa-49b0-452b-8136-157cdb6390cf\") " pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" Oct 31 14:45:43.320698 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b0ed9d-d90a-497a-8863-7605a3c89c75-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.320734 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320706 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.320765 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320735 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.320796 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320760 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2f6\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.320796 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-stats-auth\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.320852 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320816 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-snapshots\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.320852 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmnq\" (UniqueName: \"kubernetes.io/projected/e3ce5b4f-032b-4791-bc7d-0e5916aba070-kube-api-access-fjmnq\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.320906 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320863 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-service-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.320906 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.320967 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.320967 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320935 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blljj\" (UniqueName: \"kubernetes.io/projected/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-kube-api-access-blljj\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.321026 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.321026 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.320995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqw8\" (UniqueName: \"kubernetes.io/projected/8bffb79a-a3c4-4fc8-b742-148342ad6c35-kube-api-access-4nqw8\") pod \"kas-connection-checker-685895c7c9-gq2qd\" (UID: \"8bffb79a-a3c4-4fc8-b742-148342ad6c35\") " pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" Oct 31 14:45:43.321110 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321032 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/f9259fbe-036b-4629-a947-2dcd24185c78-kube-api-access-pxh9b\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.321110 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-tmp\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.321110 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b0ed9d-d90a-497a-8863-7605a3c89c75-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.321247 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321115 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9259fbe-036b-4629-a947-2dcd24185c78-serving-cert\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.321247 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321138 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.321247 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-default-certificate\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.321247 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5cwp\" (UniqueName: \"kubernetes.io/projected/45be3fe3-424d-4775-8203-ec94e0853bc1-kube-api-access-v5cwp\") pod \"kas-connection-checker-685895c7c9-b4frh\" (UID: \"45be3fe3-424d-4775-8203-ec94e0853bc1\") " pod="kube-system/kas-connection-checker-685895c7c9-b4frh" Oct 31 14:45:43.321247 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.321392 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.321422 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321394 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdr68\" (UniqueName: \"kubernetes.io/projected/810a09fc-7671-46ff-a0d9-222437cb00a1-kube-api-access-vdr68\") pod \"network-check-source-6f78d59f99-2p6pl\" (UID: \"810a09fc-7671-46ff-a0d9-222437cb00a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" Oct 31 14:45:43.321456 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.321540 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.321488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.344868 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.344849 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-699cdb64d9-r57rx"] Oct 31 14:45:43.346081 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.346035 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7fv54"] Oct 31 14:45:43.347227 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.347207 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678477bddd-n4c79"] Oct 31 14:45:43.358353 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.358330 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f"] Oct 31 14:45:43.366216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366198 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 14:45:43.366515 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366490 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.366613 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366529 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.366613 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366535 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 14:45:43.366705 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366629 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.366827 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366810 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.366906 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366886 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.366906 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366891 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 14:45:43.366906 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366886 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.367035 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366907 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.367035 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366916 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.367035 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366925 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 14:45:43.367035 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366935 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.367035 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366954 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.367035 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.366982 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.367035 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.367036 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.367250 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.367065 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 14:45:43.367250 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.367086 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.367250 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.367093 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.367250 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.367109 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.367367 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.367331 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.367676 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.367663 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 14:45:43.419378 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.419345 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ddb7677d5-drb9t"] Oct 31 14:45:43.422632 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.422696 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422684 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9259fbe-036b-4629-a947-2dcd24185c78-serving-cert\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.422757 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.422808 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-default-certificate\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.422850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422818 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ttx\" (UniqueName: \"kubernetes.io/projected/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-kube-api-access-27ttx\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.422850 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58930f87-6f50-486e-9ab6-b58f7aee05c2-serving-cert\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.422936 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cwp\" (UniqueName: \"kubernetes.io/projected/45be3fe3-424d-4775-8203-ec94e0853bc1-kube-api-access-v5cwp\") pod \"kas-connection-checker-685895c7c9-b4frh\" (UID: \"45be3fe3-424d-4775-8203-ec94e0853bc1\") " pod="kube-system/kas-connection-checker-685895c7c9-b4frh" Oct 31 14:45:43.422989 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-tmp-dir\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.422989 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.422953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.423073 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.423126 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdr68\" (UniqueName: \"kubernetes.io/projected/810a09fc-7671-46ff-a0d9-222437cb00a1-kube-api-access-vdr68\") pod \"network-check-source-6f78d59f99-2p6pl\" (UID: \"810a09fc-7671-46ff-a0d9-222437cb00a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" Oct 31 14:45:43.423126 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.423221 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423123 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.423221 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423151 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.423221 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.423221 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv59l\" (UniqueName: \"kubernetes.io/projected/f3b0ed9d-d90a-497a-8863-7605a3c89c75-kube-api-access-gv59l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.423400 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423236 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.423400 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423277 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58930f87-6f50-486e-9ab6-b58f7aee05c2-config\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.423400 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.423286 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:43.923268006 +0000 UTC m=+30.595951770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:43.423564 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.423564 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423543 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.423660 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423554 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.423660 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477-kube-api-access-bqplf\") pod \"volume-data-source-validator-58c9f5f46d-4lmnj\" (UID: \"a8319738-e6b9-47aa-9ceb-5f0bf4d5f477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" Oct 31 14:45:43.423660 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423620 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.423660 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfxw\" (UniqueName: \"kubernetes.io/projected/14236efa-49b0-452b-8136-157cdb6390cf-kube-api-access-rkfxw\") pod \"kas-connection-checker-685895c7c9-5p7xg\" (UID: \"14236efa-49b0-452b-8136-157cdb6390cf\") " pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" Oct 31 14:45:43.423844 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2npb\" (UniqueName: \"kubernetes.io/projected/1d447892-64c2-41a6-a155-65d0717627fd-kube-api-access-t2npb\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.423844 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b0ed9d-d90a-497a-8863-7605a3c89c75-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.423844 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.423844 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423755 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a946f2-c459-4f23-8f9a-91935eeb49b0-cert\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.423844 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.424071 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.423820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2f6\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.424071 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.424071 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424042 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-stats-auth\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.424071 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.424264 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424088 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.424264 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424110 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-snapshots\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.424264 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g799v\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.424264 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245fz\" (UniqueName: \"kubernetes.io/projected/d1a946f2-c459-4f23-8f9a-91935eeb49b0-kube-api-access-245fz\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.424264 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfksq\" (UniqueName: \"kubernetes.io/projected/4f7891ca-5583-456c-b2e4-e65723e27b16-kube-api-access-mfksq\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.424522 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmnq\" (UniqueName: \"kubernetes.io/projected/e3ce5b4f-032b-4791-bc7d-0e5916aba070-kube-api-access-fjmnq\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.424584 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-service-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.424637 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424584 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:43.424694 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.424746 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424696 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.424798 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.424798 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.424778 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.425108 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blljj\" (UniqueName: \"kubernetes.io/projected/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-kube-api-access-blljj\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.425232 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425120 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b0ed9d-d90a-497a-8863-7605a3c89c75-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.425232 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g72hw\" (UniqueName: \"kubernetes.io/projected/58930f87-6f50-486e-9ab6-b58f7aee05c2-kube-api-access-g72hw\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.425232 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425204 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c9bb56-e9d1-4328-a6a8-db47dba6268e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:43.425415 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425241 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.425415 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425243 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-config-volume\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.425415 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-config\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.425415 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.425622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425430 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.425622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqw8\" (UniqueName: \"kubernetes.io/projected/8bffb79a-a3c4-4fc8-b742-148342ad6c35-kube-api-access-4nqw8\") pod \"kas-connection-checker-685895c7c9-gq2qd\" (UID: \"8bffb79a-a3c4-4fc8-b742-148342ad6c35\") " pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" Oct 31 14:45:43.425622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425538 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-service-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.425622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/f9259fbe-036b-4629-a947-2dcd24185c78-kube-api-access-pxh9b\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.425622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d447892-64c2-41a6-a155-65d0717627fd-serving-cert\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.425622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-tmp\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.426049 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.426049 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.425678 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b0ed9d-d90a-497a-8863-7605a3c89c75-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.426382 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.426359 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-tmp\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.426749 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.426722 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-snapshots\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.428276 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.428252 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9259fbe-036b-4629-a947-2dcd24185c78-serving-cert\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.428358 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.428292 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b0ed9d-d90a-497a-8863-7605a3c89c75-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.446201 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.446178 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 14:45:43.446303 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.446284 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 14:45:43.446418 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.446391 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 14:45:43.446503 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.446421 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 14:45:43.446503 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.446427 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 14:45:43.446674 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.446659 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 14:45:43.447103 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.447087 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sptz6"] Oct 31 14:45:43.447940 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.447920 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dm4bk\"" Oct 31 14:45:43.447940 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.447938 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" Oct 31 14:45:43.448089 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.447927 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 14:45:43.448089 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.447943 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 14:45:43.448089 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.447971 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fzc6l\"" Oct 31 14:45:43.448089 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.448009 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tgszw\"" Oct 31 14:45:43.453210 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.453198 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:43.453253 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.453245 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:43.953230118 +0000 UTC m=+30.625913859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:43.453874 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.453858 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.454399 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.454380 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.456567 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.456550 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-stats-auth\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.459454 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459397 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 14:45:43.459454 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459399 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 14:45:43.459635 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459460 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 14:45:43.459635 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459488 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gsm2m\"" Oct 31 14:45:43.459635 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459600 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 14:45:43.459635 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459622 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 14:45:43.459879 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459626 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 14:45:43.459879 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459711 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sffft\"" Oct 31 14:45:43.459879 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.459756 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" Oct 31 14:45:43.460783 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.460771 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 14:45:43.465196 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.465179 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:43.465326 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.465235 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:43.965218833 +0000 UTC m=+30.637902580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:43.465950 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.465933 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-default-certificate\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.466033 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.465980 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:43.466033 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.465990 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:43.466033 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.466023 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:43.966009836 +0000 UTC m=+30.638693596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:43.467113 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.467095 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.467381 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.467364 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 14:45:43.467490 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.467453 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xknrx\"" Oct 31 14:45:43.467544 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.467493 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-89zxn\"" Oct 31 14:45:43.467544 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.467512 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t2tz4\"" Oct 31 14:45:43.468329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.468269 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.468329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.468270 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.468329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.468289 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:43.468329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.468276 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 14:45:43.468329 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.468303 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 14:45:43.468662 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.468641 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:43.473565 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.473540 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2"] Oct 31 14:45:43.473686 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.473670 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gdnkp"] Oct 31 14:45:43.474800 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.474784 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vh7rz"] Oct 31 14:45:43.476433 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.476417 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.481897 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.481876 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv59l\" (UniqueName: \"kubernetes.io/projected/f3b0ed9d-d90a-497a-8863-7605a3c89c75-kube-api-access-gv59l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.482090 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.482072 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cwp\" (UniqueName: \"kubernetes.io/projected/45be3fe3-424d-4775-8203-ec94e0853bc1-kube-api-access-v5cwp\") pod \"kas-connection-checker-685895c7c9-b4frh\" (UID: \"45be3fe3-424d-4775-8203-ec94e0853bc1\") " pod="kube-system/kas-connection-checker-685895c7c9-b4frh" Oct 31 14:45:43.485175 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.485152 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477-kube-api-access-bqplf\") pod \"volume-data-source-validator-58c9f5f46d-4lmnj\" (UID: \"a8319738-e6b9-47aa-9ceb-5f0bf4d5f477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" Oct 31 14:45:43.485317 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.485295 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2f6\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.485493 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.485454 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:43.485650 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.485629 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blljj\" (UniqueName: \"kubernetes.io/projected/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-kube-api-access-blljj\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:43.485743 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.485718 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdr68\" (UniqueName: \"kubernetes.io/projected/810a09fc-7671-46ff-a0d9-222437cb00a1-kube-api-access-vdr68\") pod \"network-check-source-6f78d59f99-2p6pl\" (UID: \"810a09fc-7671-46ff-a0d9-222437cb00a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" Oct 31 14:45:43.485810 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.485718 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmnq\" (UniqueName: \"kubernetes.io/projected/e3ce5b4f-032b-4791-bc7d-0e5916aba070-kube-api-access-fjmnq\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.485810 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.485760 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqw8\" (UniqueName: \"kubernetes.io/projected/8bffb79a-a3c4-4fc8-b742-148342ad6c35-kube-api-access-4nqw8\") pod \"kas-connection-checker-685895c7c9-gq2qd\" (UID: \"8bffb79a-a3c4-4fc8-b742-148342ad6c35\") " pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" Oct 31 14:45:43.486639 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.486620 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfxw\" (UniqueName: \"kubernetes.io/projected/14236efa-49b0-452b-8136-157cdb6390cf-kube-api-access-rkfxw\") pod \"kas-connection-checker-685895c7c9-5p7xg\" (UID: \"14236efa-49b0-452b-8136-157cdb6390cf\") " pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" Oct 31 14:45:43.487451 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.487433 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/f9259fbe-036b-4629-a947-2dcd24185c78-kube-api-access-pxh9b\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.514944 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.514916 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b4frh" Oct 31 14:45:43.520091 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.520074 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" Oct 31 14:45:43.527059 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527040 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.527118 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527077 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.527118 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.527230 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58930f87-6f50-486e-9ab6-b58f7aee05c2-config\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.527488 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527401 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2npb\" (UniqueName: \"kubernetes.io/projected/1d447892-64c2-41a6-a155-65d0717627fd-kube-api-access-t2npb\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.527488 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a946f2-c459-4f23-8f9a-91935eeb49b0-cert\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.527581 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.527581 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.527581 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527544 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.527581 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527568 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g799v\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.527779 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.527728 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:43.527779 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.527744 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:43.527779 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527745 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.527779 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527757 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58930f87-6f50-486e-9ab6-b58f7aee05c2-config\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.527969 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" Oct 31 14:45:43.527969 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.527810 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:44.027790247 +0000 UTC m=+30.700474018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:43.527969 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527939 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245fz\" (UniqueName: \"kubernetes.io/projected/d1a946f2-c459-4f23-8f9a-91935eeb49b0-kube-api-access-245fz\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.528130 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.527983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfksq\" (UniqueName: \"kubernetes.io/projected/4f7891ca-5583-456c-b2e4-e65723e27b16-kube-api-access-mfksq\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.528130 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:43.528130 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.528130 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.528314 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72hw\" (UniqueName: \"kubernetes.io/projected/58930f87-6f50-486e-9ab6-b58f7aee05c2-kube-api-access-g72hw\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.528314 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c9bb56-e9d1-4328-a6a8-db47dba6268e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:43.528314 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-config-volume\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.528314 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-config\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.528314 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528261 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.528314 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528307 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d447892-64c2-41a6-a155-65d0717627fd-serving-cert\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.528622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.528622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.528622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ttx\" (UniqueName: \"kubernetes.io/projected/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-kube-api-access-27ttx\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.528622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58930f87-6f50-486e-9ab6-b58f7aee05c2-serving-cert\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.528622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528485 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-tmp-dir\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.528622 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528529 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.528899 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528712 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.528971 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.529061 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.529116 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:44.029100985 +0000 UTC m=+30.701784743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.529146 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-tmp-dir\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.529220 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.529225 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.529259 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:44.029246144 +0000 UTC m=+30.701929885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.529286 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:44.029266249 +0000 UTC m=+30.701950015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.529413 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-config-volume\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.529715 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-config\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.530027 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.529932 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.530920 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.530580 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c9bb56-e9d1-4328-a6a8-db47dba6268e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:43.531107 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.531083 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.531592 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.531568 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a946f2-c459-4f23-8f9a-91935eeb49b0-cert\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.532081 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.531914 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58930f87-6f50-486e-9ab6-b58f7aee05c2-serving-cert\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.532081 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.531920 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:43.532081 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.532035 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d447892-64c2-41a6-a155-65d0717627fd-serving-cert\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.532833 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.532812 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.557732 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.556312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:43.572553 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.571922 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" Oct 31 14:45:43.577376 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.577209 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" Oct 31 14:45:43.590449 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.589989 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2npb\" (UniqueName: \"kubernetes.io/projected/1d447892-64c2-41a6-a155-65d0717627fd-kube-api-access-t2npb\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.594946 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.594718 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:43.609360 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.608693 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72hw\" (UniqueName: \"kubernetes.io/projected/58930f87-6f50-486e-9ab6-b58f7aee05c2-kube-api-access-g72hw\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.609360 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.609111 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ttx\" (UniqueName: \"kubernetes.io/projected/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-kube-api-access-27ttx\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:43.620978 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.612116 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g799v\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.620978 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.612881 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfksq\" (UniqueName: \"kubernetes.io/projected/4f7891ca-5583-456c-b2e4-e65723e27b16-kube-api-access-mfksq\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:43.620978 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.615052 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245fz\" (UniqueName: \"kubernetes.io/projected/d1a946f2-c459-4f23-8f9a-91935eeb49b0-kube-api-access-245fz\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.620978 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.615060 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:43.635640 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.634874 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:43.638417 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.638389 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:43.927493 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.927455 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5p7xg"] Oct 31 14:45:43.929631 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:43.929595 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14236efa_49b0_452b_8136_157cdb6390cf.slice/crio-64f95d8d9d5672f429eb819c477114732cfcd6717cde4676997f43cd3a88cf51 WatchSource:0}: Error finding container 64f95d8d9d5672f429eb819c477114732cfcd6717cde4676997f43cd3a88cf51: Status 404 returned error can't find the container with id 64f95d8d9d5672f429eb819c477114732cfcd6717cde4676997f43cd3a88cf51 Oct 31 14:45:43.932255 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.932236 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m"] Oct 31 14:45:43.932303 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.932284 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b4frh"] Oct 31 14:45:43.934534 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:43.934515 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b0ed9d_d90a_497a_8863_7605a3c89c75.slice/crio-181af56c2ecab4bc77549171392c694098249a693bcc6f3495a8eecebbc0f630 WatchSource:0}: Error finding container 181af56c2ecab4bc77549171392c694098249a693bcc6f3495a8eecebbc0f630: Status 404 returned error can't find the container with id 181af56c2ecab4bc77549171392c694098249a693bcc6f3495a8eecebbc0f630 Oct 31 14:45:43.935451 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:43.935431 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45be3fe3_424d_4775_8203_ec94e0853bc1.slice/crio-2cc894e6a79eec79911070624718aea0097adbb3ce94275fbcecf52c7a297c8c WatchSource:0}: Error finding container 2cc894e6a79eec79911070624718aea0097adbb3ce94275fbcecf52c7a297c8c: Status 404 returned error can't find the container with id 2cc894e6a79eec79911070624718aea0097adbb3ce94275fbcecf52c7a297c8c Oct 31 14:45:43.938666 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.938649 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gq2qd"] Oct 31 14:45:43.939189 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.939170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:43.939373 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:43.939339 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:44.939315375 +0000 UTC m=+31.611999120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:43.940447 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:43.940353 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bffb79a_a3c4_4fc8_b742_148342ad6c35.slice/crio-18a1f9ebc481f5f21116b6c989e2108bb9133574baf674a20300c333fe870445 WatchSource:0}: Error finding container 18a1f9ebc481f5f21116b6c989e2108bb9133574baf674a20300c333fe870445: Status 404 returned error can't find the container with id 18a1f9ebc481f5f21116b6c989e2108bb9133574baf674a20300c333fe870445 Oct 31 14:45:43.995858 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:43.995819 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-n4mj8"] Oct 31 14:45:44.000926 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:44.000892 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9259fbe_036b_4629_a947_2dcd24185c78.slice/crio-f338d8adec0572ccb1f7ee635756a2a0910e0e67d4234e7255491b30050a41f9 WatchSource:0}: Error finding container f338d8adec0572ccb1f7ee635756a2a0910e0e67d4234e7255491b30050a41f9: Status 404 returned error can't find the container with id f338d8adec0572ccb1f7ee635756a2a0910e0e67d4234e7255491b30050a41f9 Oct 31 14:45:44.025273 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.025246 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj"] Oct 31 14:45:44.033924 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:44.033894 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8319738_e6b9_47aa_9ceb_5f0bf4d5f477.slice/crio-5695df1251919806a4698f4c9a5523720c882c9f591c47ace9bc05b9b48b67bf WatchSource:0}: Error finding container 5695df1251919806a4698f4c9a5523720c882c9f591c47ace9bc05b9b48b67bf: Status 404 returned error can't find the container with id 5695df1251919806a4698f4c9a5523720c882c9f591c47ace9bc05b9b48b67bf Oct 31 14:45:44.036119 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.035056 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gdnkp"] Oct 31 14:45:44.040010 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.039988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:44.040096 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.040036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:44.040096 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.040069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:44.040202 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.040118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:44.040202 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.040168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:44.040288 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.040203 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:44.040288 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.040257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:44.040454 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040437 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:44.040585 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040517 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:45.040498979 +0000 UTC m=+31.713182732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:44.040659 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040593 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:44.040659 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040604 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:44.040659 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040641 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:45.040628548 +0000 UTC m=+31.713312296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:44.040809 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040701 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:44.040809 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040736 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:45.040725515 +0000 UTC m=+31.713409260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:44.040809 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040789 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:44.041014 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040820 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:45.040809961 +0000 UTC m=+31.713493702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:44.041014 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040878 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:44.041014 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040890 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:44.041014 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040919 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:45.040909097 +0000 UTC m=+31.713592841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:44.041014 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.040981 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:44.041014 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.041010 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:45.041000578 +0000 UTC m=+31.713684321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:44.042443 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.041066 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:44.042443 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.041100 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:45.041089814 +0000 UTC m=+31.713773561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:44.057519 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.057487 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl"] Oct 31 14:45:44.065139 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:44.065113 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810a09fc_7671_46ff_a0d9_222437cb00a1.slice/crio-424b492c2530481ac1af02a3284b67134f881aae93d5bf15af8324ec76f01b03 WatchSource:0}: Error finding container 424b492c2530481ac1af02a3284b67134f881aae93d5bf15af8324ec76f01b03: Status 404 returned error can't find the container with id 424b492c2530481ac1af02a3284b67134f881aae93d5bf15af8324ec76f01b03 Oct 31 14:45:44.070365 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.070338 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" event={"ID":"a8319738-e6b9-47aa-9ceb-5f0bf4d5f477","Type":"ContainerStarted","Data":"5695df1251919806a4698f4c9a5523720c882c9f591c47ace9bc05b9b48b67bf"} Oct 31 14:45:44.073808 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.073780 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" event={"ID":"14236efa-49b0-452b-8136-157cdb6390cf","Type":"ContainerStarted","Data":"64f95d8d9d5672f429eb819c477114732cfcd6717cde4676997f43cd3a88cf51"} Oct 31 14:45:44.075011 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.074994 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-n4mj8" event={"ID":"f9259fbe-036b-4629-a947-2dcd24185c78","Type":"ContainerStarted","Data":"f338d8adec0572ccb1f7ee635756a2a0910e0e67d4234e7255491b30050a41f9"} Oct 31 14:45:44.076708 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.076690 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" event={"ID":"1d447892-64c2-41a6-a155-65d0717627fd","Type":"ContainerStarted","Data":"ea20e9aff63e2aa3f01ede38d3be298ba8bec3b719b3cb913085e7bcdb490f0c"} Oct 31 14:45:44.078322 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.078297 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" event={"ID":"810a09fc-7671-46ff-a0d9-222437cb00a1","Type":"ContainerStarted","Data":"424b492c2530481ac1af02a3284b67134f881aae93d5bf15af8324ec76f01b03"} Oct 31 14:45:44.080783 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.080761 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" event={"ID":"8bffb79a-a3c4-4fc8-b742-148342ad6c35","Type":"ContainerStarted","Data":"18a1f9ebc481f5f21116b6c989e2108bb9133574baf674a20300c333fe870445"} Oct 31 14:45:44.083139 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.083122 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b4frh" event={"ID":"45be3fe3-424d-4775-8203-ec94e0853bc1","Type":"ContainerStarted","Data":"2cc894e6a79eec79911070624718aea0097adbb3ce94275fbcecf52c7a297c8c"} Oct 31 14:45:44.084684 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.084665 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" event={"ID":"f3b0ed9d-d90a-497a-8863-7605a3c89c75","Type":"ContainerStarted","Data":"181af56c2ecab4bc77549171392c694098249a693bcc6f3495a8eecebbc0f630"} Oct 31 14:45:44.102388 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.102367 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2"] Oct 31 14:45:44.103610 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.103585 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vh7rz"] Oct 31 14:45:44.105253 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:44.105224 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58930f87_6f50_486e_9ab6_b58f7aee05c2.slice/crio-f88ca9a6802b383bff0cd39f3439a4bbd07537d05786d43e7363d11bf0a136cf WatchSource:0}: Error finding container f88ca9a6802b383bff0cd39f3439a4bbd07537d05786d43e7363d11bf0a136cf: Status 404 returned error can't find the container with id f88ca9a6802b383bff0cd39f3439a4bbd07537d05786d43e7363d11bf0a136cf Oct 31 14:45:44.106091 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:44.106071 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a946f2_c459_4f23_8f9a_91935eeb49b0.slice/crio-b38baa97cde77639f747efaaa758979271a327a15ac1654e16074d54c9bac629 WatchSource:0}: Error finding container b38baa97cde77639f747efaaa758979271a327a15ac1654e16074d54c9bac629: Status 404 returned error can't find the container with id b38baa97cde77639f747efaaa758979271a327a15ac1654e16074d54c9bac629 Oct 31 14:45:44.217108 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.217073 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:44.453345 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.453309 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:44.841978 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.841934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:44.841978 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.841961 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:44.842217 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.842114 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:44.896015 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.895952 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 14:45:44.896015 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.895996 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" Oct 31 14:45:44.896198 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.896127 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 14:45:44.896198 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.896137 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" Oct 31 14:45:44.896198 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.896183 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" Oct 31 14:45:44.949036 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:44.949013 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:44.949383 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:44.949140 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:46.949126667 +0000 UTC m=+33.621810421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:45.050338 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.050295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:45.050530 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.050354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:45.050530 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.050394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:45.050530 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.050443 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:45.050696 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.050558 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:45.050696 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.050597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:45.050696 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.050633 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:45.050823 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.050772 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:45.050876 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.050850 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:47.050812637 +0000 UTC m=+33.723496393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:45.051037 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051015 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:45.051154 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051072 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:47.051055495 +0000 UTC m=+33.723739256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:45.051154 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051075 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:45.051154 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051086 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:45.051154 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051096 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:45.051154 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051144 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:47.051127239 +0000 UTC m=+33.723811012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:45.051154 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051149 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:45.051431 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051161 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:45.051431 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051161 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:47.051152147 +0000 UTC m=+33.723835889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:45.051431 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051192 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:45.051431 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051205 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:47.051193157 +0000 UTC m=+33.723876898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:45.051431 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051227 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:47.051213509 +0000 UTC m=+33.723897257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:45.051431 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051231 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:45.051431 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:45.051268 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:47.051257895 +0000 UTC m=+33.723941641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:45.089517 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.089461 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" event={"ID":"8bffb79a-a3c4-4fc8-b742-148342ad6c35","Type":"ContainerStarted","Data":"3aac9ec85045df4d03f45099bdfc7d2b167bd7d7f5ac943c1c3ee9b5323c0722"} Oct 31 14:45:45.091791 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.091728 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b4frh" event={"ID":"45be3fe3-424d-4775-8203-ec94e0853bc1","Type":"ContainerStarted","Data":"ccda27dec5a68eb07628864892370bbbe9f1aec24275a2fe7659138c7b7195c9"} Oct 31 14:45:45.093939 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.093884 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vh7rz" event={"ID":"d1a946f2-c459-4f23-8f9a-91935eeb49b0","Type":"ContainerStarted","Data":"b38baa97cde77639f747efaaa758979271a327a15ac1654e16074d54c9bac629"} Oct 31 14:45:45.096359 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.096006 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" event={"ID":"14236efa-49b0-452b-8136-157cdb6390cf","Type":"ContainerStarted","Data":"78af884c17b3294551977fa3ce66cd15fe25da0ae2747cb791827a555dd5af93"} Oct 31 14:45:45.098091 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.098040 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" event={"ID":"58930f87-6f50-486e-9ab6-b58f7aee05c2","Type":"ContainerStarted","Data":"f88ca9a6802b383bff0cd39f3439a4bbd07537d05786d43e7363d11bf0a136cf"} Oct 31 14:45:45.406892 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.406832 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" podStartSLOduration=433.406813028 podStartE2EDuration="7m13.406813028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:45.405577938 +0000 UTC m=+32.078261701" watchObservedRunningTime="2026-10-31 14:45:45.406813028 +0000 UTC m=+32.079496786" Oct 31 14:45:45.696136 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.696044 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" podStartSLOduration=433.696028483 podStartE2EDuration="7m13.696028483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:45.695604778 +0000 UTC m=+32.368288542" watchObservedRunningTime="2026-10-31 14:45:45.696028483 +0000 UTC m=+32.368712236" Oct 31 14:45:45.696307 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:45.696255 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-b4frh" podStartSLOduration=433.696248689 podStartE2EDuration="7m13.696248689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:45.52581016 +0000 UTC m=+32.198493926" watchObservedRunningTime="2026-10-31 14:45:45.696248689 +0000 UTC m=+32.368932451" Oct 31 14:45:46.770315 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:46.770262 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:46.770814 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:46.770407 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:46.770814 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:46.770630 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:46.770814 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:46.770700 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:18.770680553 +0000 UTC m=+65.443364312 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : secret "metrics-daemon-secret" not found Oct 31 14:45:46.774550 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:46.774497 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:46.871838 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:46.871302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:46.875910 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:46.875865 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:46.971987 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:46.971952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:46.972269 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:46.972250 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.97222991 +0000 UTC m=+37.644913658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:47.013791 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.013551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:47.057593 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.057563 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.072951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.072987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.073009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.073037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.073056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.073075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.073104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073264 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073665 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073726 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.073707696 +0000 UTC m=+37.746391442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073795 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073805 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073835 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.073824915 +0000 UTC m=+37.746508660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073890 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073910 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.073903835 +0000 UTC m=+37.746587576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:47.074216 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073957 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:47.075386 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.073985 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.073975819 +0000 UTC m=+37.746659566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:47.075386 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.074039 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:47.075386 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.074576 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.074557579 +0000 UTC m=+37.747241326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:47.075386 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.074593 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:47.075386 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.074633 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.074622887 +0000 UTC m=+37.747306643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:47.075386 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.074049 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:47.075386 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:47.074918 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.074897749 +0000 UTC m=+37.747581494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:47.127345 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.127219 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" event={"ID":"a8319738-e6b9-47aa-9ceb-5f0bf4d5f477","Type":"ContainerStarted","Data":"82d086a73719da85cf8b382bed063cd74b15c48bd918e2b0f7d3e40ad90dc74c"} Oct 31 14:45:47.247134 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.247077 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" podStartSLOduration=402.356779101 podStartE2EDuration="6m45.247055744s" totalImagesPullingTime="2.890276643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:47.245720122 +0000 UTC m=+33.918403886" watchObservedRunningTime="2026-10-31 14:45:47.247055744 +0000 UTC m=+33.919739509" Oct 31 14:45:47.348770 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.348686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:47.428981 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.428607 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jrfkr"] Oct 31 14:45:47.431563 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:47.431536 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07979459_61ec_49fc_9696_2c985d7ad790.slice/crio-03ea037ca1ce947fe2285c55d394145855b34d69f283a848a4f14a855445f80d WatchSource:0}: Error finding container 03ea037ca1ce947fe2285c55d394145855b34d69f283a848a4f14a855445f80d: Status 404 returned error can't find the container with id 03ea037ca1ce947fe2285c55d394145855b34d69f283a848a4f14a855445f80d Oct 31 14:45:47.487793 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.487719 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:47.536675 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.536636 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h78q8"] Oct 31 14:45:47.550020 ip-10-0-131-60 kubenswrapper[2570]: W1031 14:45:47.549994 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eefaca9_ad88_44ac_be6b_363497696b72.slice/crio-e722ff7b431b9095caffd2ac4b87631f44739853f937d32c30ea429a05589b5c WatchSource:0}: Error finding container e722ff7b431b9095caffd2ac4b87631f44739853f937d32c30ea429a05589b5c: Status 404 returned error can't find the container with id e722ff7b431b9095caffd2ac4b87631f44739853f937d32c30ea429a05589b5c Oct 31 14:45:47.615983 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:47.615879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:48.136003 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:48.135647 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h78q8" event={"ID":"8eefaca9-ad88-44ac-be6b-363497696b72","Type":"ContainerStarted","Data":"e722ff7b431b9095caffd2ac4b87631f44739853f937d32c30ea429a05589b5c"} Oct 31 14:45:48.137928 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:48.137897 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jrfkr" event={"ID":"07979459-61ec-49fc-9696-2c985d7ad790","Type":"ContainerStarted","Data":"03ea037ca1ce947fe2285c55d394145855b34d69f283a848a4f14a855445f80d"} Oct 31 14:45:48.679760 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:48.679728 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:48.876184 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:48.876137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:49.148229 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:49.148184 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-n4mj8" event={"ID":"f9259fbe-036b-4629-a947-2dcd24185c78","Type":"ContainerStarted","Data":"6d9305c38f7284dbc5dc3201d85197a3188a71ee91eaf0564e4f4a27c7ce5d9b"} Oct 31 14:45:49.280531 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:49.278583 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-n4mj8" podStartSLOduration=402.575865173 podStartE2EDuration="6m47.27856695s" totalImagesPullingTime="4.702701777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:49.278386907 +0000 UTC m=+35.951070696" watchObservedRunningTime="2026-10-31 14:45:49.27856695 +0000 UTC m=+35.951250714" Oct 31 14:45:49.988804 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:49.988667 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:50.170067 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.169976 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:50.177524 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.176648 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/0.log" Oct 31 14:45:50.177524 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.176687 2570 generic.go:356] "Generic (PLEG): container finished" podID="1d447892-64c2-41a6-a155-65d0717627fd" containerID="23112c5392f5d6ecccea7d028973b84a4aa23fe0f6f2126ab1c6588e463fd891" exitCode=255 Oct 31 14:45:50.177524 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.177094 2570 scope.go:122] "RemoveContainer" containerID="23112c5392f5d6ecccea7d028973b84a4aa23fe0f6f2126ab1c6588e463fd891" Oct 31 14:45:50.177524 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.177416 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" event={"ID":"1d447892-64c2-41a6-a155-65d0717627fd","Type":"ContainerDied","Data":"23112c5392f5d6ecccea7d028973b84a4aa23fe0f6f2126ab1c6588e463fd891"} Oct 31 14:45:50.177794 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.177738 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-clmnr"] Oct 31 14:45:50.208658 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.208622 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" event={"ID":"f3b0ed9d-d90a-497a-8863-7605a3c89c75","Type":"ContainerStarted","Data":"8bc1c92793439050e5bd14fd0dfdbab7305df888bc3d43fe9545b3590b4ff213"} Oct 31 14:45:50.208794 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.208668 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" event={"ID":"58930f87-6f50-486e-9ab6-b58f7aee05c2","Type":"ContainerStarted","Data":"c9125bd1ae51819fe329b8f5b34c307b3e44db08998e17bbad4528820e263843"} Oct 31 14:45:50.208794 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.208769 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.215855 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.215779 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clmnr"] Oct 31 14:45:50.315336 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.315305 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.315521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.315364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-data-volume\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.315521 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.315404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.315641 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.315575 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xcg\" (UniqueName: \"kubernetes.io/projected/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-api-access-s8xcg\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.315690 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.315667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-crio-socket\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.331628 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.331567 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 14:45:50.331628 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.331611 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 14:45:50.332512 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.331871 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" Oct 31 14:45:50.375436 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.375406 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:50.417018 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.416815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-data-volume\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.417018 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.416869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.417018 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.416931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xcg\" (UniqueName: \"kubernetes.io/projected/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-api-access-s8xcg\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.417018 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.416975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-crio-socket\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.417339 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.417030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.417339 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.417132 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.417339 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.417184 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.91716574 +0000 UTC m=+37.589849494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.417593 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.417426 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-data-volume\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.418738 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.418716 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:50.418906 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.418780 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.918761853 +0000 UTC m=+37.591445596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:50.422749 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.422554 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-crio-socket\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.477086 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.477054 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xcg\" (UniqueName: \"kubernetes.io/projected/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-api-access-s8xcg\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.513755 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.513679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:50.550225 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.550130 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" podStartSLOduration=402.837469654 podStartE2EDuration="6m48.550094421s" totalImagesPullingTime="5.712624767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:50.538146623 +0000 UTC m=+37.210830387" watchObservedRunningTime="2026-10-31 14:45:50.550094421 +0000 UTC m=+37.222778188" Oct 31 14:45:50.668086 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.668028 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" podStartSLOduration=403.237469377 podStartE2EDuration="6m48.668008369s" totalImagesPullingTime="5.430538992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:50.667611102 +0000 UTC m=+37.340294867" watchObservedRunningTime="2026-10-31 14:45:50.668008369 +0000 UTC m=+37.340692133" Oct 31 14:45:50.938554 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.923506 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.938554 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.923606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:50.938554 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.923693 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.938554 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.923712 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:50.938554 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.923784 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.923762791 +0000 UTC m=+38.596446553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.938554 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:50.923808 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.923796694 +0000 UTC m=+38.596480435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:50.987645 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:50.987556 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:51.025047 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.025014 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:51.025187 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.025173 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.025156687 +0000 UTC m=+45.697840428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:51.125869 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.125659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:51.125869 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.125738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:51.125869 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.125782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:51.125869 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.125791 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:51.125869 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.125855 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.125837606 +0000 UTC m=+45.798521363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:51.125869 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.125868 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.125878 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.125919 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.125903733 +0000 UTC m=+45.798587475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.125941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.125981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.126009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126031 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126043 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.126055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126080 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.126067192 +0000 UTC m=+45.798750950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126125 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126139 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.126132771 +0000 UTC m=+45.798816511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126163 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.126151424 +0000 UTC m=+45.798835171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126203 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126232 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126238 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.126228164 +0000 UTC m=+45.798911951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:51.126351 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126243 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:51.127302 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.126280 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.126266576 +0000 UTC m=+45.798950317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:51.127302 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.126675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:51.192372 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.192287 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/1.log" Oct 31 14:45:51.192818 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.192800 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/0.log" Oct 31 14:45:51.192863 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.192836 2570 generic.go:356] "Generic (PLEG): container finished" podID="1d447892-64c2-41a6-a155-65d0717627fd" containerID="806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103" exitCode=255 Oct 31 14:45:51.192940 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.192916 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" event={"ID":"1d447892-64c2-41a6-a155-65d0717627fd","Type":"ContainerDied","Data":"806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103"} Oct 31 14:45:51.193066 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.193006 2570 scope.go:122] "RemoveContainer" containerID="23112c5392f5d6ecccea7d028973b84a4aa23fe0f6f2126ab1c6588e463fd891" Oct 31 14:45:51.193277 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.193260 2570 scope.go:122] "RemoveContainer" containerID="806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103" Oct 31 14:45:51.193544 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.193518 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gdnkp_openshift-console-operator(1d447892-64c2-41a6-a155-65d0717627fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" podUID="1d447892-64c2-41a6-a155-65d0717627fd" Oct 31 14:45:51.195249 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.195228 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" event={"ID":"810a09fc-7671-46ff-a0d9-222437cb00a1","Type":"ContainerStarted","Data":"eef6ecb4799ea6a1830c5a34c4ba8ee5b2a0251cd47b09bbb7373d28f71e2ef9"} Oct 31 14:45:51.197293 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.197269 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vh7rz" event={"ID":"d1a946f2-c459-4f23-8f9a-91935eeb49b0","Type":"ContainerStarted","Data":"bb1a2fcb6cc357cd92145568be50f7d63840116da78f025124386d3f6b7a5158"} Oct 31 14:45:51.289710 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.289671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:51.415184 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.415148 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:51.544539 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.544482 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vh7rz" podStartSLOduration=2.507680624 podStartE2EDuration="8.544447434s" totalImagesPullingTime="6.03676681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:51.543250695 +0000 UTC m=+38.215934458" watchObservedRunningTime="2026-10-31 14:45:51.544447434 +0000 UTC m=+38.217131199" Oct 31 14:45:51.544959 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.544927 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" podStartSLOduration=400.617912447 podStartE2EDuration="6m47.544914421s" totalImagesPullingTime="6.927001974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:51.426624094 +0000 UTC m=+38.099307857" watchObservedRunningTime="2026-10-31 14:45:51.544914421 +0000 UTC m=+38.217598184" Oct 31 14:45:51.557193 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.557164 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:51.687432 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.687390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:51.818587 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.817924 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:51.933126 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.933096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:51.933818 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:51.933174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:51.933818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.933281 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:51.933818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.933323 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:51.933818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.933349 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:53.933328926 +0000 UTC m=+40.606012668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:51.933818 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:51.933533 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:53.933366853 +0000 UTC m=+40.606050612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:52.001384 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:52.001352 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:52.144018 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:52.143936 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:52.202050 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:52.202020 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/1.log" Oct 31 14:45:52.202532 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:52.202441 2570 scope.go:122] "RemoveContainer" containerID="806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103" Oct 31 14:45:52.202741 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:52.202705 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gdnkp_openshift-console-operator(1d447892-64c2-41a6-a155-65d0717627fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" podUID="1d447892-64c2-41a6-a155-65d0717627fd" Oct 31 14:45:52.205200 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:52.204833 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jrfkr" event={"ID":"07979459-61ec-49fc-9696-2c985d7ad790","Type":"ContainerStarted","Data":"6ffd005efb8f27cf244b354231937bc06db56315a7c1ba219a558c7867ca46ef"} Oct 31 14:45:52.436038 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:52.435930 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jrfkr" podStartSLOduration=34.537607132 podStartE2EDuration="38.435909815s" totalImagesPullingTime="3.898302683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:52.43415242 +0000 UTC m=+39.106836183" watchObservedRunningTime="2026-10-31 14:45:52.435909815 +0000 UTC m=+39.108593582" Oct 31 14:45:53.016101 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.016065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:53.170214 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.170173 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:53.208057 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.208028 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:53.586835 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.586801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:53.596534 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.596509 2570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:53.596673 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.596570 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:53.596982 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.596968 2570 scope.go:122] "RemoveContainer" containerID="806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103" Oct 31 14:45:53.597207 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:53.597179 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gdnkp_openshift-console-operator(1d447892-64c2-41a6-a155-65d0717627fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" podUID="1d447892-64c2-41a6-a155-65d0717627fd" Oct 31 14:45:53.952789 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.952713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:53.952789 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:53.952769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:53.953003 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:53.952866 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:53.953003 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:53.952937 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.952919366 +0000 UTC m=+44.625603125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:53.953003 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:53.952870 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:53.953003 ip-10-0-131-60 kubenswrapper[2570]: E1031 14:45:53.953002 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.952991139 +0000 UTC m=+44.625674881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:54.212971 ip-10-0-131-60 kubenswrapper[2570]: I1031 14:45:54.212932 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h78q8" event={"ID":"8eefaca9-ad88-44ac-be6b-363497696b72","Type":"ContainerStarted","Data":"0d678de8c68be3023909eaf4b55c9ec03a4dc53f59bbf7d6e6972aaaaf4074f4"} Oct 31 14:45:54.312261 ip-10-0-131-60 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 14:45:54.323843 ip-10-0-131-60 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 14:45:54.324008 ip-10-0-131-60 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 14:45:54.324068 ip-10-0-131-60 systemd[1]: kubelet.service: Consumed 3.030s CPU time. Oct 31 14:45:54.339097 ip-10-0-131-60 systemd[1]: Starting Kubernetes Kubelet... Oct 31 14:45:54.408581 ip-10-0-131-60 kubenswrapper[4865]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:54.408581 ip-10-0-131-60 kubenswrapper[4865]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 14:45:54.408581 ip-10-0-131-60 kubenswrapper[4865]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:54.408581 ip-10-0-131-60 kubenswrapper[4865]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:54.410679 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410588 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.410679 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410679 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.410679 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410683 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410687 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410690 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410693 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410695 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410698 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410701 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410703 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410706 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410708 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410711 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410713 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410716 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410719 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410721 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410724 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410726 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410729 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410731 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410734 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.410783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410737 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410740 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410746 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410749 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410752 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410755 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410757 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410760 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410764 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410768 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410770 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410773 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410776 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410778 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410781 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410784 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410788 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410790 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410793 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410796 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.411276 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410799 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410801 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410803 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410806 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410808 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410811 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410813 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410816 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410819 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410822 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410825 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410827 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410830 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410832 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410835 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410837 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410840 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410842 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410845 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410847 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.411790 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410849 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410852 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410854 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410856 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410860 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410863 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410865 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410868 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410870 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410873 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410876 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410879 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410883 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410886 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410888 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410891 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410893 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410896 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410898 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.412284 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410901 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410904 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410907 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410909 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410912 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410914 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410917 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410920 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410923 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410926 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410928 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410931 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410933 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410935 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410938 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410940 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410943 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410945 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410948 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.424912 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410951 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410954 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410956 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410959 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410961 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.410964 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411746 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411753 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411756 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411758 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411761 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411765 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411769 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411772 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411775 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411777 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411780 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411783 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411786 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.425570 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411788 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411791 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411794 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411796 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411799 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411802 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411804 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411807 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411810 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411812 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411815 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411817 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411821 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411824 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411827 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411829 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411832 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411834 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411837 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.426526 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411839 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411842 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411844 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411846 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411849 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411851 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411854 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411856 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411858 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411861 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411863 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411867 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411869 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411872 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411876 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411880 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411883 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411885 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411888 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411890 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.427202 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411893 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411895 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411898 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411900 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411903 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411908 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411911 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411913 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411916 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411918 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411921 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411923 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411925 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411928 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411930 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411933 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411935 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411937 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411940 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.427853 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411943 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411946 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411948 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411951 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411953 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411956 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411959 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411961 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411964 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411967 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411970 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411973 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411976 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411978 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411980 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411983 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411985 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411988 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411991 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411994 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.428385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411996 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.411999 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412001 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412004 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412006 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412008 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412011 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412014 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412016 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412018 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412022 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412025 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412027 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412030 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412032 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412535 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412541 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412546 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412549 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.428917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412552 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412555 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412557 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412560 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412562 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412565 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412567 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412570 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412573 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412575 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412578 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412581 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412584 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412586 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412589 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412591 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412593 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412596 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412599 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412601 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.429388 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412606 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412609 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412611 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412614 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412616 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412619 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412621 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412624 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412626 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412628 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412631 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412634 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412637 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412639 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412642 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412644 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412647 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412650 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412652 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412655 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.429894 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412658 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412660 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412663 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412666 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412668 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412670 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412673 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412675 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412678 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412680 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412683 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412686 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412690 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412693 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412696 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412699 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412702 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412705 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412719 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.430383 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412722 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412726 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412730 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412734 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412738 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412743 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412747 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412751 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412754 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412757 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412760 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412762 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412764 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412767 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412769 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412772 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412775 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412778 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412780 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412783 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.430872 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412785 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412788 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412791 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412793 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412796 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412800 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412802 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412805 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412809 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412812 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412815 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412817 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412820 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412822 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412825 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412827 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412829 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412832 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412835 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412838 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.431530 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412840 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412843 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.412846 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412937 4865 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412945 4865 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412951 4865 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412955 4865 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412960 4865 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412963 4865 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412967 4865 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412971 4865 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412974 4865 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412977 4865 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412980 4865 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412984 4865 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412987 4865 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412990 4865 flags.go:64] FLAG: --cgroup-root="" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412994 4865 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412997 4865 flags.go:64] FLAG: --client-ca-file="" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.412999 4865 flags.go:64] FLAG: --cloud-provider="external" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413002 4865 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413007 4865 flags.go:64] FLAG: --cluster-domain="" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413009 4865 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413012 4865 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 14:45:54.432330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413015 4865 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413019 4865 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413023 4865 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413025 4865 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413028 4865 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413032 4865 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413035 4865 flags.go:64] FLAG: --contention-profiling="false" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413038 4865 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413041 4865 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413044 4865 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413047 4865 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413051 4865 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413054 4865 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413057 4865 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413060 4865 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413064 4865 flags.go:64] FLAG: --enable-server="true" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413067 4865 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413071 4865 flags.go:64] FLAG: --event-burst="100" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413074 4865 flags.go:64] FLAG: --event-qps="50" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413077 4865 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413080 4865 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413082 4865 flags.go:64] FLAG: --eviction-hard="" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413087 4865 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413090 4865 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413093 4865 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 14:45:54.433114 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413098 4865 flags.go:64] FLAG: --eviction-soft="" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413101 4865 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413104 4865 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413107 4865 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413110 4865 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413113 4865 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413116 4865 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413118 4865 flags.go:64] FLAG: --feature-gates="" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413125 4865 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413128 4865 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413131 4865 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413134 4865 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413137 4865 flags.go:64] FLAG: --healthz-port="10248" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413140 4865 flags.go:64] FLAG: --help="false" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413144 4865 flags.go:64] FLAG: --hostname-override="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413147 4865 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413150 4865 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413153 4865 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413156 4865 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413159 4865 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413162 4865 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413165 4865 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413169 4865 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 14:45:54.433751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413172 4865 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413175 4865 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413178 4865 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413181 4865 flags.go:64] FLAG: --kube-reserved="" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413184 4865 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413187 4865 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413190 4865 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413193 4865 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413195 4865 flags.go:64] FLAG: --lock-file="" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413200 4865 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413203 4865 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413206 4865 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413212 4865 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413214 4865 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413217 4865 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413220 4865 flags.go:64] FLAG: --logging-format="text" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413223 4865 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413227 4865 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413230 4865 flags.go:64] FLAG: --manifest-url="" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413232 4865 flags.go:64] FLAG: --manifest-url-header="" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413237 4865 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413240 4865 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413243 4865 flags.go:64] FLAG: --max-pods="110" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413246 4865 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413250 4865 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 14:45:54.434369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413253 4865 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413256 4865 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413259 4865 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413262 4865 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413264 4865 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413271 4865 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413276 4865 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413279 4865 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413282 4865 flags.go:64] FLAG: --pod-cidr="" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413285 4865 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413288 4865 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413291 4865 flags.go:64] FLAG: --pods-per-core="0" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413293 4865 flags.go:64] FLAG: --port="10250" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413296 4865 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413299 4865 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-06e1420c05f26bdac" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413302 4865 flags.go:64] FLAG: --qos-reserved="" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413306 4865 flags.go:64] FLAG: --read-only-port="10255" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413309 4865 flags.go:64] FLAG: --register-node="true" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413313 4865 flags.go:64] FLAG: --register-with-taints="" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413316 4865 flags.go:64] FLAG: --registry-burst="10" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413319 4865 flags.go:64] FLAG: --registry-qps="5" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413322 4865 flags.go:64] FLAG: --reserved-cpus="" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413324 4865 flags.go:64] FLAG: --reserved-memory="" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413328 4865 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413331 4865 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413334 4865 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 14:45:54.435000 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413337 4865 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413339 4865 flags.go:64] FLAG: --runonce="false" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413342 4865 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413345 4865 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413348 4865 flags.go:64] FLAG: --seccomp-default="false" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413351 4865 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413354 4865 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413357 4865 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413360 4865 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413363 4865 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413365 4865 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413369 4865 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413372 4865 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413375 4865 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413378 4865 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413381 4865 flags.go:64] FLAG: --system-cgroups="" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413384 4865 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413389 4865 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413392 4865 flags.go:64] FLAG: --tls-cert-file="" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413395 4865 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413399 4865 flags.go:64] FLAG: --tls-min-version="" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413402 4865 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413405 4865 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413408 4865 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413411 4865 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 14:45:54.435659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413414 4865 flags.go:64] FLAG: --v="2" Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413418 4865 flags.go:64] FLAG: --version="false" Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413422 4865 flags.go:64] FLAG: --vmodule="" Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413426 4865 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413429 4865 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413573 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413577 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413580 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413582 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413585 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413588 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413591 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413593 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413595 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413598 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413601 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413604 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413606 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413609 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413613 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.436264 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413616 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413618 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413621 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413623 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413626 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413628 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413630 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413633 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413635 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413639 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413641 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413644 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413646 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413649 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413651 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413654 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413656 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413659 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413661 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413664 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.436791 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413666 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413668 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413671 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413674 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413676 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413679 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413681 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413683 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413686 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413689 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413691 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413696 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413698 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413701 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413703 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413706 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413708 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413711 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413713 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.437296 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413716 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413718 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413722 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413725 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413727 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413730 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413732 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413735 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413737 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413741 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413745 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413748 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413750 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413753 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413755 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413758 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413760 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413763 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413765 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.437783 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413768 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413770 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413773 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413776 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413778 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413782 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413785 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413787 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413790 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413792 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413795 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413797 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413800 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413802 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413805 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413808 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413811 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413813 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413817 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.438250 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413820 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413824 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413827 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413829 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413832 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413834 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413837 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413839 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413841 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413844 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413846 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413849 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413851 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413854 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.413878 4865 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:54.438729 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413913 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413918 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413922 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413927 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413930 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413933 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413936 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413938 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413941 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413944 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413946 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413949 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413951 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413954 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413957 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413960 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413962 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413965 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413969 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.439104 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413973 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413976 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413979 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413982 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413985 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413987 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413990 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413993 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413995 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.413998 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414000 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414003 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414006 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414009 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414012 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414015 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414020 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414022 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414025 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414027 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.439643 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414030 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414032 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414035 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414037 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414039 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414042 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414044 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414049 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414053 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414056 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414058 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414060 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414063 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414066 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414068 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414071 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414073 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414075 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414078 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.440237 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414080 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414083 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414085 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414088 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414090 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414093 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414095 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414098 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414100 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414104 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414107 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414109 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414112 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414114 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414116 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414119 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414122 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414124 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414127 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414129 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.440724 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414133 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414136 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414138 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414141 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414143 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414146 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414149 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414151 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414154 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414156 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414159 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414161 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414163 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414175 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414177 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414180 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414182 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414185 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414187 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414190 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.441254 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414193 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414197 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414199 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414202 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414205 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414208 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414211 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.414213 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.419334 4865 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.419350 4865 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.419374 4865 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.419379 4865 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419439 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419445 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419448 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419451 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419454 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419458 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419462 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.441906 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419479 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419483 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419485 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419488 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419491 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419494 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419496 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419499 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419502 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419506 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419513 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419517 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419521 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419524 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419527 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419535 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419537 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419540 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419543 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.442385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419546 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419548 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419551 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419554 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419556 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419559 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419561 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419564 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419567 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419569 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419572 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419574 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419577 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419579 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419581 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419584 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419586 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419589 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419591 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419593 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.442862 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419601 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419604 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419607 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419609 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419612 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419614 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419617 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419619 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419623 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419626 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419628 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419631 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419633 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419636 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419638 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419641 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419643 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419645 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419648 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.443343 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419651 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419653 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419656 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419659 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419661 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419663 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419666 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419668 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419671 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419673 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419676 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419678 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419681 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419683 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419691 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419694 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419697 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419699 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419702 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.443818 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419704 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419707 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419710 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419713 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419716 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419718 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419721 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419723 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419726 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419728 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419731 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419733 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419736 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419738 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419741 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419743 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419746 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419748 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419751 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419753 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.444273 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419755 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419758 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.419784 4865 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419938 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419942 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419945 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419948 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419952 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419957 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419960 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419963 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419966 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419968 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419971 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419973 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419977 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.444778 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419980 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419983 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419985 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419988 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419990 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419993 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419995 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.419998 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420001 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420003 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420006 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420009 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420011 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420013 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420017 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420021 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420025 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420028 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420031 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.445187 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420034 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420036 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420038 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420041 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420043 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420047 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420050 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420053 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420055 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420057 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420060 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420062 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420066 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420069 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420071 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420074 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420076 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420079 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420081 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420084 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.445788 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420086 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420089 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420091 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420093 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420114 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420118 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420121 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420132 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420136 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420139 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420142 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420145 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420148 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420150 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420153 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420155 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420157 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420165 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420168 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420171 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.446294 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420173 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420176 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420178 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420181 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420184 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420194 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420197 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420200 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420203 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420206 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420208 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420211 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420213 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420216 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420218 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420221 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420223 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420226 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420228 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.446799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420231 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420234 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420236 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420238 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420241 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420243 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420246 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420248 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420250 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420253 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420257 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420260 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420262 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420265 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420268 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.447325 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.420290 4865 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420324 4865 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420328 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420332 4865 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420334 4865 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420337 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420340 4865 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420343 4865 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420345 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420348 4865 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420350 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420353 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420355 4865 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420357 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420360 4865 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420363 4865 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420365 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420368 4865 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420370 4865 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:54.447857 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420372 4865 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420375 4865 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420377 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420380 4865 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420382 4865 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420385 4865 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420387 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420389 4865 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420393 4865 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420396 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420398 4865 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420401 4865 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420404 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420407 4865 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420409 4865 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420412 4865 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420416 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420418 4865 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420421 4865 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420423 4865 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:54.448336 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420425 4865 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420428 4865 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420430 4865 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420433 4865 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420435 4865 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420438 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420440 4865 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420444 4865 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420447 4865 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420451 4865 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420454 4865 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420457 4865 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420460 4865 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420463 4865 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420481 4865 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420483 4865 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420486 4865 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420489 4865 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420491 4865 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:54.449143 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420493 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:54.448978 ip-10-0-131-60 systemd[1]: Started Kubernetes Kubelet. Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420497 4865 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420500 4865 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420502 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420505 4865 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420508 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420511 4865 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420514 4865 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420517 4865 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420520 4865 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420523 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420526 4865 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420528 4865 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420531 4865 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420533 4865 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420536 4865 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420538 4865 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420541 4865 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420543 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420546 4865 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:54.450116 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420548 4865 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420551 4865 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420553 4865 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420556 4865 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420558 4865 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420561 4865 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420563 4865 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420566 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420568 4865 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420571 4865 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420573 4865 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420575 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420578 4865 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420581 4865 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420584 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420586 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420589 4865 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420591 4865 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420593 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420596 4865 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:54.451071 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420599 4865 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420603 4865 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420605 4865 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420608 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420610 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420613 4865 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420615 4865 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420619 4865 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:45:54.420622 4865 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.420678 4865 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.421654 4865 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.421709 4865 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.421899 4865 server.go:1014] "Starting client certificate rotation" Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.422037 4865 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.422081 4865 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 14:40:13 +0000 UTC" deadline="2028-06-30 07:49:20.90450242 +0000 UTC" Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.422116 4865 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14585h3m26.48238874s" Oct 31 14:45:54.451932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.422414 4865 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:54.452601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.423496 4865 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:54.452601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.424849 4865 log.go:25] "Validated CRI v1 runtime API" Oct 31 14:45:54.452601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.426959 4865 log.go:25] "Validated CRI v1 image API" Oct 31 14:45:54.452601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.427763 4865 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 14:45:54.452601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.429690 4865 fs.go:135] Filesystem UUIDs: map[0985ecc2-f783-4eb3-ae78-4fe5692cec0c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d329977-038f-4bbc-9c40-4e9b2741fa3b:/dev/nvme0n1p4] Oct 31 14:45:54.452878 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.429709 4865 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03ea037ca1ce947fe2285c55d394145855b34d69f283a848a4f14a855445f80d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03ea037ca1ce947fe2285c55d394145855b34d69f283a848a4f14a855445f80d/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/181af56c2ecab4bc77549171392c694098249a693bcc6f3495a8eecebbc0f630/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/181af56c2ecab4bc77549171392c694098249a693bcc6f3495a8eecebbc0f630/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18a1f9ebc481f5f21116b6c989e2108bb9133574baf674a20300c333fe870445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18a1f9ebc481f5f21116b6c989e2108bb9133574baf674a20300c333fe870445/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c2419a8387d77a20bfd3f9be4e8c2baddb018d8745b9eb0608b86b7fd8537b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c2419a8387d77a20bfd3f9be4e8c2baddb018d8745b9eb0608b86b7fd8537b0/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cc894e6a79eec79911070624718aea0097adbb3ce94275fbcecf52c7a297c8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cc894e6a79eec79911070624718aea0097adbb3ce94275fbcecf52c7a297c8c/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/424b492c2530481ac1af02a3284b67134f881aae93d5bf15af8324ec76f01b03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/424b492c2530481ac1af02a3284b67134f881aae93d5bf15af8324ec76f01b03/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ec2aec1f1e48c4041a1650aa97ba69790060028d15d401763b71e00acf483c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ec2aec1f1e48c4041a1650aa97ba69790060028d15d401763b71e00acf483c4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5695df1251919806a4698f4c9a5523720c882c9f591c47ace9bc05b9b48b67bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5695df1251919806a4698f4c9a5523720c882c9f591c47ace9bc05b9b48b67bf/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64f95d8d9d5672f429eb819c477114732cfcd6717cde4676997f43cd3a88cf51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64f95d8d9d5672f429eb819c477114732cfcd6717cde4676997f43cd3a88cf51/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77c985bb831b42140dac8051b5e9e9a3af7a488317b0461ab3e4045b6c27b5d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77c985bb831b42140dac8051b5e9e9a3af7a488317b0461ab3e4045b6c27b5d9/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c7ab459cc5dcc8cc6006449b48842a0fd69b5f2d5bed6e112431d56446f0049/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c7ab459cc5dcc8cc6006449b48842a0fd69b5f2d5bed6e112431d56446f0049/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b38baa97cde77639f747efaaa758979271a327a15ac1654e16074d54c9bac629/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b38baa97cde77639f747efaaa758979271a327a15ac1654e16074d54c9bac629/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7b29bf60ed9315b159e46541de7ec1007b913a2aaceea4c97757f98c4f09569/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7b29bf60ed9315b159e46541de7ec1007b913a2aaceea4c97757f98c4f09569/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e722ff7b431b9095caffd2ac4b87631f44739853f937d32c30ea429a05589b5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e722ff7b431b9095caffd2ac4b87631f44739853f937d32c30ea429a05589b5c/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea20e9aff63e2aa3f01ede38d3be298ba8bec3b719b3cb913085e7bcdb490f0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea20e9aff63e2aa3f01ede38d3be298ba8bec3b719b3cb913085e7bcdb490f0c/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f338d8adec0572ccb1f7ee635756a2a0910e0e67d4234e7255491b30050a41f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f338d8adec0572ccb1f7ee635756a2a0910e0e67d4234e7255491b30050a41f9/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f83dc6a977e4125f7941bbaf63776685e36b83f5cd9ea128b9020f92b9632244/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f83dc6a977e4125f7941bbaf63776685e36b83f5cd9ea128b9020f92b9632244/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f88ca9a6802b383bff0cd39f3439a4bbd07537d05786d43e7363d11bf0a136cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f88ca9a6802b383bff0cd39f3439a4bbd07537d05786d43e7363d11bf0a136cf/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbe864b6740ca9d006f3fb0168700f5bd1d64a30329ae07ac4975440678b1a87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbe864b6740ca9d006f3fb0168700f5bd1d64a30329ae07ac4975440678b1a87/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd4a293ac95cf4a9bbabe7be1f8ecf27627505e72ddfbff13a8eb635fe7ffcde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd4a293ac95cf4a9bbabe7be1f8ecf27627505e72ddfbff13a8eb635fe7ffcde/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07979459-61ec-49fc-9696-2c985d7ad790/volumes/kubernetes.io~projected/kube-api-access-75kjc:{mountpoint:/var/lib/kubelet/pods/07979459-61ec-49fc-9696-2c985d7ad790/volumes/kubernetes.io~projected/kube-api-access-75kjc major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a562fc8-6e09-49d5-9dcf-3188700438eb/volumes/kubernetes.io~projected/kube-api-access-rp9tt:{mountpoint:/var/lib/kubelet/pods/0a562fc8-6e09-49d5-9dcf-3188700438eb/volumes/kubernetes.io~projected/kube-api-access-rp9tt major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bafcc61-160a-4f2a-9081-29bd47254408/volumes/kubernetes.io~projected/kube-api-access-jfktw:{mountpoint:/var/lib/kubelet/pods/0bafcc61-160a-4f2a-9081-29bd47254408/volumes/kubernetes.io~projected/kube-api-access-jfktw major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14236efa-49b0-452b-8136-157cdb6390cf/volumes/kubernetes.io~projected/kube-api-access-rkfxw:{mountpoint:/var/lib/kubelet/pods/14236efa-49b0-452b-8136-157cdb6390cf/volumes/kubernetes.io~projected/kube-api-access-rkfxw major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d447892-64c2-41a6-a155-65d0717627fd/volumes/kubernetes.io~projected/kube-api-access-t2npb:{mountpoint:/var/lib/kubelet/pods/1d447892-64c2-41a6-a155-65d0717627fd/volumes/kubernetes.io~projected/kube-api-access-t2npb major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d447892-64c2-41a6-a155-65d0717627fd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1d447892-64c2-41a6-a155-65d0717627fd/volumes/kubernetes.io~secret/serving-cert major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40101b79-243a-4844-be24-6ecd1b07f67a/volumes/kubernetes.io~projected/kube-api-access-c8cff:{mountpoint:/var/lib/kubelet/pods/40101b79-243a-4844-be24-6ecd1b07f67a/volumes/kubernetes.io~projected/kube-api-access-c8cff major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45be3fe3-424d-4775-8203-ec94e0853bc1/volumes/kubernetes.io~projected/kube-api-access-v5cwp:{mountpoint:/var/lib/kubelet/pods/45be3fe3-424d-4775-8203-ec94e0853bc1/volumes/kubernetes.io~projected/kube-api-access-v5cwp major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f7891ca-5583-456c-b2e4-e65723e27b16/volumes/kubernetes.io~projected/kube-api-access-mfksq:{mountpoint:/var/lib/kubelet/pods/4f7891ca-5583-456c-b2e4-e65723e27b16/volumes/kubernetes.io~projected/kube-api-access-mfksq major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58930f87-6f50-486e-9ab6-b58f7aee05c2/volumes/kubernetes.io~projected/kube-api-access-g72hw:{mountpoint:/var/lib/kubelet/pods/58930f87-6f50-486e-9ab6-b58f7aee05c2/volumes/kubernetes.io~projected/kube-api-access-g72hw major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58930f87-6f50-486e-9ab6-b58f7aee05c2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/58930f87-6f50-486e-9ab6-b58f7aee05c2/volumes/kubernetes.io~secret/serving-cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~projected/kube-api-access-pg9tp:{mountpoint:/var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~projected/kube-api-access-pg9tp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad/volumes/kubernetes.io~projected/kube-api-access-blljj:{mountpoint:/var/lib/kubelet/pods/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad/volumes/kubernetes.io~projected/kube-api-access-blljj major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/788d00c1-792d-47ea-bcd3-cfee09779724/volumes/kubernetes.io~projected/kube-api-access-n8bmx:{mountpoint:/var/lib/kubelet/pods/788d00c1-792d-47ea-bcd3-cfee09779724/volumes/kubernetes.io~projected/kube-api-access-n8bmx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d91b481-174f-488b-97d8-d9a1c716a71d/volumes/kubernetes.io~projected/kube-api-access-8qwjg:{mountpoint:/var/lib/kubelet/pods/7d91b481-174f-488b-97d8-d9a1c716a71d/volumes/kubernetes.io~projected/kube-api-access-8qwjg major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/810a09fc-7671-46ff-a0d9-222437cb00a1/volumes/kubernetes.io~projected/kube-api-access-vdr68:{mountpoint:/var/lib/kubelet/pods/810a09fc-7671-46ff-a0d9-222437cb00a1/volumes/kubernetes.io~projected/kube-api-access-vdr68 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~projected/kube-api-access-2w2f6:{mountpoint:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~projected/kube-api-access-2w2f6 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bffb79a-a3c4-4fc8-b742-148342ad6c35/volumes/kubernetes.io~projected/kube-api-access-4nqw8:{mountpoint:/var/lib/kubelet/pods/8bffb79a-a3c4-4fc8-b742-148342ad6c35/volumes/kubernetes.io~projected/kube-api-access-4nqw8 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eefaca9-ad88-44ac-be6b-363497696b72/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8eefaca9-ad88-44ac-be6b-363497696b72/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/959b8335-545a-4af4-a81b-8e4efee557e8/volumes/kubernetes.io~projected/kube-api-access-d8tdm:{mountpoint:/var/lib/kubelet/pods/959b8335-545a-4af4-a81b-8e4efee557e8/volumes/kubernetes.io~projected/kube-api-access-d8tdm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477/volumes/kubernetes.io~projected/kube-api-access-bqplf:{mountpoint:/var/lib/kubelet/pods/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477/volumes/kubernetes.io~projected/kube-api-access-bqplf major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a93ffb8f-f9d5-4663-a591-d94fd7ca296f/volumes/kubernetes.io~projected/kube-api-access-s8xcg:{mountpoint:/var/lib/kubelet/pods/a93ffb8f-f9d5-4663-a591-d94fd7ca296f/volumes/kubernetes.io~projected/kube-api-access-s8xcg major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8/volumes/kubernetes.io~projected/kube-api-access-27ttx:{mountpoint:/var/lib/kubelet/pods/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8/volumes/kubernetes.io~projected/kube-api-access-27ttx major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc/volumes/kubernetes.io~projected/kube-api-access-jgdlz:{mountpoint:/var/lib/kubelet/pods/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc/volumes/kubernetes.io~projected/kube-api-access-jgdlz major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1a946f2-c459-4f23-8f9a-91935eeb49b0/volumes/kubernetes.io~projected/kube-api-access-245fz:{mountpoint:/var/lib/kubelet/pods/d1a946f2-c459-4f23-8f9a-91935eeb49b0/volumes/kubernetes.io~projected/kube-api-access-245fz major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1a946f2-c459-4f23-8f9a-91935eeb49b0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d1a946f2-c459-4f23-8f9a-91935eeb49b0/volumes/kubernetes.io~secret/cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~projected/kube-api-access-g799v:{mountpoint:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~projected/kube-api-access-g799v major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~projected/kube-api-access-fjmnq:{mountpoint:/var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~projected/kube-api-access-fjmnq major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volumes/kubernetes.io~projected/kube-api-access-kttct:{mountpoint:/var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volumes/kubernetes.io~projected/kube-api-access-kttct major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b0ed9d-d90a-497a-8863-7605a3c89c75/volumes/kubernetes.io~projected/kube-api-access-gv59l:{mountpoint:/var/lib/kubelet/pods/f3b0ed9d-d90a-497a-8863-7605a3c89c75/volumes/kubernetes.io~projected/kube-api-access-gv59l major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b0ed9d-d90a-497a-8863-7605a3c89c75/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f3b0ed9d-d90a-497a-8863-7605a3c89c75/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9259fbe-036b-4629-a947-2dcd24185c78/volumes/kubernetes.io~projected/kube-api-access-pxh9b:{mountpoint:/var/lib/kubelet/pods/f9259fbe-036b-4629-a947-2dcd24185c78/volumes/kubernetes.io~projected/kube-api-access-pxh9b major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9259fbe-036b-4629-a947-2dcd24185c78/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f9259fbe-036b-4629-a947-2dcd24185c78/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fb3e352e85bd8de862fbf5f710f7032836de3b6ecca9cf56babf3053947f5ab2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d7d2c6fe6a83944841b8ab6ee33a69c20a012b0445fcc1f0b6f388e7d40f73cd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/20a4120ffeb79bf259a86e02b58df61de638f4544521d37643cf2cb0de060bef/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/6a73741f1b50e7322eae2613a557ed35a2788efaf8cdb4db5c22e3250a2f6e8e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/035b3b3532cf5075383847f253df5606c81b551e2ec0996b2622ed944da3051a/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/d558ec353256edb3548b39bfd687cb015fff77602b692d693030748a13c2887c/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cd9013e803d2695d3e5417ff4bf36815dfc6b552c2053d0287d90a672d1fb4e8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e29693928e010e39b752f45ee95d0a7cc52aa85dce3b249bfab3e4a1b705248a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/85cf3822b9cb0c525c866beb4829506b945c2b0d3f48f59c2c5df15cabfe09dc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/264ee0e4db94ab65169e585d9088979f073b96940e442c8507f01c1395ef74af/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1b8178db069f2db5a43a0bd6584dfbc7235ac29ec293377ea47f2836c961283b/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/7aeb1531edc3c48b583622f58a4ad1e035ea499f8637791d235d948f5fa1ba58/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7435bb72d66c58d9adfbcca1b63abb10c05789337027c9f76f846af035d0ec7e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f6df976cba2fdc488f5ca74ffd115565660cba4aeaedb227c899367e718f122e/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ec666684607972273ed31b39956e34bc3c68849b1d0a75fec1d1fca38240ab9e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/48708d37bd1471d9accda93fd776bd5a46caf8c985b22c486b1a1965e3f59f27/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4596d205ce98605c5bee80020704e29dc19829e3ca55447e09a3fb8cbe72712d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3031c6a5ecea8f413b91a91b486528e70f7b738c6a50294decf99720b264ed9e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c10279b2ac4597ab274039674e90a13cd20b74c6d1c59bae34b34a9294b794d9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/627e88b2deebd558f34fba3aeafd0806385b7e04c1e8b0a46c3aa7d551e87add/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/cff70772cebd65360aa041c6be53fdf323461abf68af3566b2a04f77a89d09fd/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9889c02b120b9db9c3cbfbea24160e22a8c228adf76d5466cfa2e8672283ae07/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/3eda77ce0facdc54a8e7f266af9589ca61cf8e8ae2632ae4238f04bcb4060466/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/e77f40f90aed160b532e5a459fad0aa4c5bbd875d92888c9dd17c45b77089ab5/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/9752d7f885f6da8ee7750e4e5ca533b2899dcb92f76645cf5a9fc49ab31bca85/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9b436d15d0dd4f23c932018e177622b45ec14b5cac830170eccb5cbab18c6c21/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/158cc2e8609c320439df23af00383312568f2de89045e9fb83be389ce4451454/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/838a99a3d651c8e9b8a5517d163978a1ba4db31d8278743baa1964a7ba382093/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/afed6e0a27cc9fe911cb43e9815bef6889013e5e741ec978aca6b8917b3d36e4/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/d21eb87c6808e21b41b0498018da5934901d9c01e7140a9b6b45548f3f0b6140/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/5887b1268a2be28389fc57a3ff5dbf777e63cb232fe3c0f3370256c237af2225/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/fbe0d09fca417b41c731fc1cba86aa89693e90968daf06cbcbc0decae56bb68d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6d59e2d9ebf3dce41b9fdcc24cbf3c2832b39f7613440ca83a84dd87523d35a8/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/392773e71f5aef9883522ecd7210e32bc1027db991cd4b58fe2f706bb925d646/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/61b4613c1c0b4d4a10d669a2489fca94fb6034cbce21b5c2b76782c56b2c028c/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/6b005af9606b7e169ab58a1b7066626a5350555cda8ecc99344a5595c466adb3/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/a5275bda7eb4f3c6ebf8f6fcda943c8cb990646433feb29d8f40d04946c7c1d1/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/5d3092d175f0d78eae8cda4e57422cc60ad910298a5b21344c8f626a00cb0064/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/67714a4afab7f815105fcdd382ff996d2980cc69781d60764082f370e38b5139/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/943914c32f241e86165dc8154284b08a724b2d2a4c2f1fcb3172526ecea4c67e/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/2451d85257ca10e70e650058ae640e6db14832962d0426ecce444efa90f6ce70/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/0f6689ebefb1988d5bf34525236ec2a9fe895139b15b8f4664d6c2caace046b5/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/7d860ade4f605a7985658600056c049995fc48810ebbb450f9eb1896b33e4a46/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/6b2a0d61c4af33993bbc3099a0ac2b63751511d93535a04cec1d0fae3d9c6e74/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/ae623f5a1048863ae3ce33809178ace58d9c65bd8be8861d385b13a9af14d4a4/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/d1537f1d9211a9f56abc201548562c35d06e33dbbc8aae22ed47c2b90811e99a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/18a569bc7f366521b40fe3ec3f414f9080500f03d6c45d9fc60dc5325d60aba8/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c6e681c1bdde60203ea88897e77d280af01b6929aacb4cf3e57553d29fe1b642/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3aa38169fba46367542a55defae6466e2c9184430a3378a205feda88c74d709d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/bc7543578361253720d3827234df58138b9a70a50fbe6fb272177cab16232ea1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/bf987a2425c891fd6a109063e727164979c43d9254dfb5c4efe6d4d2134866a7/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1f8f25cd8afef3501b5cea306a318bd8b7a1900f81b9273fb57becb24bdb52cf/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/5052fe01ecfdea42653e84466f2ec542f769b5f205b3202513fc96b105e10948/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a008e7f20108da5874459e77b0b5ddfb7761a319605c827f3a436418d1b4fa08/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.438340 4865 manager.go:217] Machine: {Timestamp:2026-10-31 14:45:54.437880773 +0000 UTC m=+0.057432176 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3152391 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a59a41a61da2f39bc909af4a42d2 SystemUUID:ec24a59a-41a6-1da2-f39b-c909af4a42d2 BootID:4fcd75dc-20b9-494b-a326-7c8b3322a412 Filesystems:[{Device:/var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7b29bf60ed9315b159e46541de7ec1007b913a2aaceea4c97757f98c4f09569/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/424b492c2530481ac1af02a3284b67134f881aae93d5bf15af8324ec76f01b03/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/07979459-61ec-49fc-9696-2c985d7ad790/volumes/kubernetes.io~projected/kube-api-access-75kjc DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbe864b6740ca9d006f3fb0168700f5bd1d64a30329ae07ac4975440678b1a87/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14236efa-49b0-452b-8136-157cdb6390cf/volumes/kubernetes.io~projected/kube-api-access-rkfxw DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77c985bb831b42140dac8051b5e9e9a3af7a488317b0461ab3e4045b6c27b5d9/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58930f87-6f50-486e-9ab6-b58f7aee05c2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f338d8adec0572ccb1f7ee635756a2a0910e0e67d4234e7255491b30050a41f9/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/788d00c1-792d-47ea-bcd3-cfee09779724/volumes/kubernetes.io~projected/kube-api-access-n8bmx DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volumes/kubernetes.io~projected/kube-api-access-kttct DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9259fbe-036b-4629-a947-2dcd24185c78/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cc894e6a79eec79911070624718aea0097adbb3ce94275fbcecf52c7a297c8c/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f1152975-ff11-49bf-b390-022434aae2a0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~projected/kube-api-access-2w2f6 DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1d447892-64c2-41a6-a155-65d0717627fd/volumes/kubernetes.io~projected/kube-api-access-t2npb DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b0ed9d-d90a-497a-8863-7605a3c89c75/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/40101b79-243a-4844-be24-6ecd1b07f67a/volumes/kubernetes.io~projected/kube-api-access-c8cff DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/181af56c2ecab4bc77549171392c694098249a693bcc6f3495a8eecebbc0f630/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~projected/kube-api-access-g799v DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd4a293ac95cf4a9bbabe7be1f8ecf27627505e72ddfbff13a8eb635fe7ffcde/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bffb79a-a3c4-4fc8-b742-148342ad6c35/volumes/kubernetes.io~projected/kube-api-access-4nqw8 DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8/volumes/kubernetes.io~projected/kube-api-access-27ttx DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f7891ca-5583-456c-b2e4-e65723e27b16/volumes/kubernetes.io~projected/kube-api-access-mfksq DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b38baa97cde77639f747efaaa758979271a327a15ac1654e16074d54c9bac629/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc/volumes/kubernetes.io~projected/kube-api-access-jgdlz DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea20e9aff63e2aa3f01ede38d3be298ba8bec3b719b3cb913085e7bcdb490f0c/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f88ca9a6802b383bff0cd39f3439a4bbd07537d05786d43e7363d11bf0a136cf/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03ea037ca1ce947fe2285c55d394145855b34d69f283a848a4f14a855445f80d/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b0ed9d-d90a-497a-8863-7605a3c89c75/volumes/kubernetes.io~projected/kube-api-access-gv59l DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad/volumes/kubernetes.io~projected/kube-api-access-blljj DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8eefaca9-ad88-44ac-be6b-363497696b72/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c7ab459cc5dcc8cc6006449b48842a0fd69b5f2d5bed6e112431d56446f0049/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64f95d8d9d5672f429eb819c477114732cfcd6717cde4676997f43cd3a88cf51/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/810a09fc-7671-46ff-a0d9-222437cb00a1/volumes/kubernetes.io~projected/kube-api-access-vdr68 DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9259fbe-036b-4629-a947-2dcd24185c78/volumes/kubernetes.io~projected/kube-api-access-pxh9b DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45be3fe3-424d-4775-8203-ec94e0853bc1/volumes/kubernetes.io~projected/kube-api-access-v5cwp DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5695df1251919806a4698f4c9a5523720c882c9f591c47ace9bc05b9b48b67bf/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7d91b481-174f-488b-97d8-d9a1c716a71d/volumes/kubernetes.io~projected/kube-api-access-8qwjg DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a562fc8-6e09-49d5-9dcf-3188700438eb/volumes/kubernetes.io~projected/kube-api-access-rp9tt DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/959b8335-545a-4af4-a81b-8e4efee557e8/volumes/kubernetes.io~projected/kube-api-access-d8tdm DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~projected/kube-api-access-fjmnq DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a93ffb8f-f9d5-4663-a591-d94fd7ca296f/volumes/kubernetes.io~projected/kube-api-access-s8xcg DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c2419a8387d77a20bfd3f9be4e8c2baddb018d8745b9eb0608b86b7fd8537b0/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477/volumes/kubernetes.io~projected/kube-api-access-bqplf DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e722ff7b431b9095caffd2ac4b87631f44739853f937d32c30ea429a05589b5c/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0bafcc61-160a-4f2a-9081-29bd47254408/volumes/kubernetes.io~projected/kube-api-access-jfktw DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58e34a13-1454-4b53-9667-23944e457c8d/volumes/kubernetes.io~projected/kube-api-access-pg9tp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1a946f2-c459-4f23-8f9a-91935eeb49b0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18a1f9ebc481f5f21116b6c989e2108bb9133574baf674a20300c333fe870445/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58930f87-6f50-486e-9ab6-b58f7aee05c2/volumes/kubernetes.io~projected/kube-api-access-g72hw DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f83dc6a977e4125f7941bbaf63776685e36b83f5cd9ea128b9020f92b9632244/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ec2aec1f1e48c4041a1650aa97ba69790060028d15d401763b71e00acf483c4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ce5b4f-032b-4791-bc7d-0e5916aba070/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1d447892-64c2-41a6-a155-65d0717627fd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1a946f2-c459-4f23-8f9a-91935eeb49b0/volumes/kubernetes.io~projected/kube-api-access-245fz DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03ea037ca1ce947 MacAddress:f6:6c:fd:34:85:0f Speed:10000 Mtu:8901} {Name:181af56c2ecab4b MacAddress:62:36:b4:d2:6c:22 Speed:10000 Mtu:8901} {Name:18a1f9ebc481f5f MacAddress:62:31:e8:f0:16:08 Speed:10000 Mtu:8901} {Name:2cc894e6a79eec7 MacAddress:6e:ca:22:aa:f1:7e Speed:10000 Mtu:8901} {Name:424b492c2530481 MacAddress:6a:2b:6c:61:75:61 Speed:10000 Mtu:8901} {Name:5695df125191980 MacAddress:c2:bc:97:bd:2f:c1 Speed:10000 Mtu:8901} {Name:64f95d8d9d5672f MacAddress:2a:a0:2b:a2:56:9e Speed:10000 Mtu:8901} {Name:b38baa97cde7763 MacAddress:26:9a:47:eb:80:8c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:4f:79:bb:d1:67 Speed:0 Mtu:9001} {Name:br-int MacAddress:56:fe:1a:f7:bc:6d Speed:0 Mtu:8901} {Name:e722ff7b431b909 MacAddress:46:27:9b:4b:75:e1 Speed:10000 Mtu:8901} {Name:ea20e9aff63e2aa MacAddress:ca:6c:d0:a0:c4:b3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:4f:79:bb:d1:67 Speed:0 Mtu:9001} {Name:f338d8adec0572c MacAddress:f2:29:b8:61:54:6e Speed:10000 Mtu:8901} {Name:f88ca9a6802b383 MacAddress:a2:10:11:55:ab:0f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7a:a8:a3:5a:5a:b2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:80:57:04:39:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.438730 4865 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.438773 4865 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439063 4865 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439082 4865 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-60.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439199 4865 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439206 4865 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439216 4865 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439229 4865 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439250 4865 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439445 4865 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439576 4865 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439610 4865 kubelet.go:493] "Attempting to sync node with API server" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439624 4865 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439634 4865 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439643 4865 kubelet.go:405] "Adding apiserver pod source" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.439651 4865 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.440729 4865 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.440743 4865 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.441404 4865 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444191 4865 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444201 4865 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444294 4865 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444509 4865 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444536 4865 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444628 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444638 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444644 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444662 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444671 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444680 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444688 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444697 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444705 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444713 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444725 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444740 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444770 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.444775 4865 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.448250 4865 server.go:1263] "Started kubelet" Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.448508 4865 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.448761 4865 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 14:45:54.453179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.449546 4865 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 14:45:54.455284 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.449551 4865 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 14:45:54.455284 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.452535 4865 server.go:317] "Adding debug handlers to kubelet server" Oct 31 14:45:54.456301 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.456283 4865 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 14:45:54.456412 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.456368 4865 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 14:45:54.457271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.456525 4865 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:14 +0000 UTC" deadline="2028-04-11 18:50:07.468044843 +0000 UTC" Oct 31 14:45:54.457271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.456550 4865 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12676h4m13.01149838s" Oct 31 14:45:54.457271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.456598 4865 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 14:45:54.457271 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.456598 4865 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" Oct 31 14:45:54.457271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.456608 4865 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 14:45:54.457271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.456617 4865 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 14:45:54.460020 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460003 4865 factory.go:153] Registering CRI-O factory Oct 31 14:45:54.460110 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460025 4865 factory.go:223] Registration of the crio container factory successfully Oct 31 14:45:54.460110 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460089 4865 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 14:45:54.460110 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460100 4865 factory.go:55] Registering systemd factory Oct 31 14:45:54.460110 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460109 4865 factory.go:223] Registration of the systemd container factory successfully Oct 31 14:45:54.460307 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460126 4865 factory.go:103] Registering Raw factory Oct 31 14:45:54.460307 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460140 4865 manager.go:1199] Started watching for new ooms in manager Oct 31 14:45:54.460511 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460494 4865 manager.go:319] Starting recovery of all containers Oct 31 14:45:54.461028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.460974 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bafcc61-160a-4f2a-9081-29bd47254408" volumeName="kubernetes.io/configmap/0bafcc61-160a-4f2a-9081-29bd47254408-serviceca" seLinuxMountContext="" Oct 31 14:45:54.461116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461042 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d447892-64c2-41a6-a155-65d0717627fd" volumeName="kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-trusted-ca" seLinuxMountContext="" Oct 31 14:45:54.461116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461067 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" volumeName="kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v" seLinuxMountContext="" Oct 31 14:45:54.461116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461080 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f7891ca-5583-456c-b2e4-e65723e27b16" volumeName="kubernetes.io/configmap/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-config" seLinuxMountContext="" Oct 31 14:45:54.461116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461092 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e34a13-1454-4b53-9667-23944e457c8d" volumeName="kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-etc-tuned" seLinuxMountContext="" Oct 31 14:45:54.461116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461105 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788d00c1-792d-47ea-bcd3-cfee09779724" volumeName="kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-cni-binary-copy" seLinuxMountContext="" Oct 31 14:45:54.461116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461116 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8340f44a-41ba-4624-9bc4-a7c8873ab788" volumeName="kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461128 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a93ffb8f-f9d5-4663-a591-d94fd7ca296f" volumeName="kubernetes.io/empty-dir/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-data-volume" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461154 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b0ed9d-d90a-497a-8863-7605a3c89c75" volumeName="kubernetes.io/projected/f3b0ed9d-d90a-497a-8863-7605a3c89c75-kube-api-access-gv59l" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461168 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d447892-64c2-41a6-a155-65d0717627fd" volumeName="kubernetes.io/secret/1d447892-64c2-41a6-a155-65d0717627fd-serving-cert" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461180 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="810a09fc-7671-46ff-a0d9-222437cb00a1" volumeName="kubernetes.io/projected/810a09fc-7671-46ff-a0d9-222437cb00a1-kube-api-access-vdr68" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461194 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8340f44a-41ba-4624-9bc4-a7c8873ab788" volumeName="kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461206 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ce5b4f-032b-4791-bc7d-0e5916aba070" volumeName="kubernetes.io/projected/e3ce5b4f-032b-4791-bc7d-0e5916aba070-kube-api-access-fjmnq" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461219 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07979459-61ec-49fc-9696-2c985d7ad790" volumeName="kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461247 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a562fc8-6e09-49d5-9dcf-3188700438eb" volumeName="kubernetes.io/projected/0a562fc8-6e09-49d5-9dcf-3188700438eb-kube-api-access-rp9tt" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461260 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45be3fe3-424d-4775-8203-ec94e0853bc1" volumeName="kubernetes.io/projected/45be3fe3-424d-4775-8203-ec94e0853bc1-kube-api-access-v5cwp" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461273 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8340f44a-41ba-4624-9bc4-a7c8873ab788" volumeName="kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461285 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83c9bb56-e9d1-4328-a6a8-db47dba6268e" volumeName="kubernetes.io/configmap/83c9bb56-e9d1-4328-a6a8-db47dba6268e-nginx-conf" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461297 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="959b8335-545a-4af4-a81b-8e4efee557e8" volumeName="kubernetes.io/empty-dir/959b8335-545a-4af4-a81b-8e4efee557e8-tmp-dir" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461310 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="959b8335-545a-4af4-a81b-8e4efee557e8" volumeName="kubernetes.io/projected/959b8335-545a-4af4-a81b-8e4efee557e8-kube-api-access-d8tdm" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461320 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" volumeName="kubernetes.io/projected/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-kube-api-access-jgdlz" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461332 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e4782a-cfd3-4b3e-9d37-01c836d0e0ad" volumeName="kubernetes.io/configmap/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-telemetry-config" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461345 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d91b481-174f-488b-97d8-d9a1c716a71d" volumeName="kubernetes.io/projected/7d91b481-174f-488b-97d8-d9a1c716a71d-kube-api-access-8qwjg" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461356 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8340f44a-41ba-4624-9bc4-a7c8873ab788" volumeName="kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461368 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eefaca9-ad88-44ac-be6b-363497696b72" volumeName="kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461381 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a93ffb8f-f9d5-4663-a591-d94fd7ca296f" volumeName="kubernetes.io/projected/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-api-access-s8xcg" seLinuxMountContext="" Oct 31 14:45:54.461397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461405 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1a946f2-c459-4f23-8f9a-91935eeb49b0" volumeName="kubernetes.io/secret/d1a946f2-c459-4f23-8f9a-91935eeb49b0-cert" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461418 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" volumeName="kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461454 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9259fbe-036b-4629-a947-2dcd24185c78" volumeName="kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-tmp" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461488 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58930f87-6f50-486e-9ab6-b58f7aee05c2" volumeName="kubernetes.io/projected/58930f87-6f50-486e-9ab6-b58f7aee05c2-kube-api-access-g72hw" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461503 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788d00c1-792d-47ea-bcd3-cfee09779724" volumeName="kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-multus-daemon-config" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461538 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788d00c1-792d-47ea-bcd3-cfee09779724" volumeName="kubernetes.io/projected/788d00c1-792d-47ea-bcd3-cfee09779724-kube-api-access-n8bmx" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461561 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8319738-e6b9-47aa-9ceb-5f0bf4d5f477" volumeName="kubernetes.io/projected/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477-kube-api-access-bqplf" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461574 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9259fbe-036b-4629-a947-2dcd24185c78" volumeName="kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-service-ca-bundle" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461591 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9259fbe-036b-4629-a947-2dcd24185c78" volumeName="kubernetes.io/secret/f9259fbe-036b-4629-a947-2dcd24185c78-serving-cert" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461617 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8" volumeName="kubernetes.io/configmap/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-config-volume" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461630 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8" volumeName="kubernetes.io/empty-dir/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-tmp-dir" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461641 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" volumeName="kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461664 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9259fbe-036b-4629-a947-2dcd24185c78" volumeName="kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-trusted-ca-bundle" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461679 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14236efa-49b0-452b-8136-157cdb6390cf" volumeName="kubernetes.io/projected/14236efa-49b0-452b-8136-157cdb6390cf-kube-api-access-rkfxw" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461726 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1a946f2-c459-4f23-8f9a-91935eeb49b0" volumeName="kubernetes.io/projected/d1a946f2-c459-4f23-8f9a-91935eeb49b0-kube-api-access-245fz" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461749 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bafcc61-160a-4f2a-9081-29bd47254408" volumeName="kubernetes.io/projected/0bafcc61-160a-4f2a-9081-29bd47254408-kube-api-access-jfktw" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461793 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e34a13-1454-4b53-9667-23944e457c8d" volumeName="kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-tmp" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461806 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8340f44a-41ba-4624-9bc4-a7c8873ab788" volumeName="kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461849 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" volumeName="kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461862 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d447892-64c2-41a6-a155-65d0717627fd" volumeName="kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-config" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461872 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40101b79-243a-4844-be24-6ecd1b07f67a" volumeName="kubernetes.io/projected/40101b79-243a-4844-be24-6ecd1b07f67a-kube-api-access-c8cff" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461896 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e4782a-cfd3-4b3e-9d37-01c836d0e0ad" volumeName="kubernetes.io/projected/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-kube-api-access-blljj" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461912 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d91b481-174f-488b-97d8-d9a1c716a71d" volumeName="kubernetes.io/configmap/7d91b481-174f-488b-97d8-d9a1c716a71d-iptables-alerter-script" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461924 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bffb79a-a3c4-4fc8-b742-148342ad6c35" volumeName="kubernetes.io/projected/8bffb79a-a3c4-4fc8-b742-148342ad6c35-kube-api-access-4nqw8" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461954 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1152975-ff11-49bf-b390-022434aae2a0" volumeName="kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-config" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461966 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58930f87-6f50-486e-9ab6-b58f7aee05c2" volumeName="kubernetes.io/configmap/58930f87-6f50-486e-9ab6-b58f7aee05c2-config" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.461977 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58930f87-6f50-486e-9ab6-b58f7aee05c2" volumeName="kubernetes.io/secret/58930f87-6f50-486e-9ab6-b58f7aee05c2-serving-cert" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462001 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8" volumeName="kubernetes.io/projected/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-kube-api-access-27ttx" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462034 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ce5b4f-032b-4791-bc7d-0e5916aba070" volumeName="kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-stats-auth" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462048 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1152975-ff11-49bf-b390-022434aae2a0" volumeName="kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-script-lib" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462101 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9259fbe-036b-4629-a947-2dcd24185c78" volumeName="kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-snapshots" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462116 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e34a13-1454-4b53-9667-23944e457c8d" volumeName="kubernetes.io/projected/58e34a13-1454-4b53-9667-23944e457c8d-kube-api-access-pg9tp" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462138 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d447892-64c2-41a6-a155-65d0717627fd" volumeName="kubernetes.io/projected/1d447892-64c2-41a6-a155-65d0717627fd-kube-api-access-t2npb" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462199 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4" volumeName="kubernetes.io/configmap/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-konnectivity-ca" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462233 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8340f44a-41ba-4624-9bc4-a7c8873ab788" volumeName="kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462249 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" volumeName="kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462273 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" volumeName="kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462287 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" volumeName="kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462298 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1152975-ff11-49bf-b390-022434aae2a0" volumeName="kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-env-overrides" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462322 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1152975-ff11-49bf-b390-022434aae2a0" volumeName="kubernetes.io/secret/f1152975-ff11-49bf-b390-022434aae2a0-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462344 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4" volumeName="kubernetes.io/secret/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-agent-certs" seLinuxMountContext="" Oct 31 14:45:54.462344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462355 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" volumeName="kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462378 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" volumeName="kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462392 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1152975-ff11-49bf-b390-022434aae2a0" volumeName="kubernetes.io/projected/f1152975-ff11-49bf-b390-022434aae2a0-kube-api-access-kttct" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462404 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b0ed9d-d90a-497a-8863-7605a3c89c75" volumeName="kubernetes.io/configmap/f3b0ed9d-d90a-497a-8863-7605a3c89c75-config" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462427 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b0ed9d-d90a-497a-8863-7605a3c89c75" volumeName="kubernetes.io/secret/f3b0ed9d-d90a-497a-8863-7605a3c89c75-serving-cert" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462449 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9259fbe-036b-4629-a947-2dcd24185c78" volumeName="kubernetes.io/projected/f9259fbe-036b-4629-a947-2dcd24185c78-kube-api-access-pxh9b" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462460 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f7891ca-5583-456c-b2e4-e65723e27b16" volumeName="kubernetes.io/projected/4f7891ca-5583-456c-b2e4-e65723e27b16-kube-api-access-mfksq" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462511 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8340f44a-41ba-4624-9bc4-a7c8873ab788" volumeName="kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462523 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" volumeName="kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-binary-copy" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462533 4865 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ce5b4f-032b-4791-bc7d-0e5916aba070" volumeName="kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-default-certificate" seLinuxMountContext="" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462565 4865 reconstruct.go:97] "Volume reconstruction finished" Oct 31 14:45:54.464438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.462577 4865 reconciler.go:29] "Reconciler: start to sync state" Oct 31 14:45:54.470961 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.470934 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/1.log" Oct 31 14:45:54.475203 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.475183 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/1.log" Oct 31 14:45:54.484504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.484487 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:45:54.486529 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.486459 4865 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 14:45:54.488220 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.488169 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:45:54.488308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.488272 4865 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 14:45:54.488308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.488290 4865 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 14:45:54.488308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.488309 4865 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 14:45:54.488493 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.488353 4865 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 14:45:54.501318 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.501276 4865 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="de8df5b72f058179ad9fbb3018757db39dbf7ccc8ef20d28c6e9f7dad5bb69f4" exitCode=0 Oct 31 14:45:54.501318 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.501318 4865 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="fb11d93c1a46bba05e02eb362ec047fcc1acd18a45df7d7b6559ba6af3eb8666" exitCode=0 Oct 31 14:45:54.501452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.501330 4865 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="56eb5cd9c8408fd838cabca31c8a9d22056e1d933b80291a21a49c6e60e8e19a" exitCode=0 Oct 31 14:45:54.501452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.501339 4865 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="933182eebc1e2ef656158d3169faa88c62fa64b0515449b17d39c4506ffa195b" exitCode=0 Oct 31 14:45:54.501452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.501347 4865 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="914b9da3849499dd2fde2790ea44705408493239bd957330142f45f4563d581c" exitCode=0 Oct 31 14:45:54.501452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.501360 4865 generic.go:356] "Generic (PLEG): container finished" podID="befbecfb-5e0b-4d8b-8e13-54c7394ba5bc" containerID="5633decd426dafc0266c4b44f0f4dd0747ca56df60a2f64b76d9a79eeac22e52" exitCode=0 Oct 31 14:45:54.503331 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.503309 4865 generic.go:356] "Generic (PLEG): container finished" podID="3e9c00f3a8edeac7e2ac58fb07ea319e" containerID="aa8b5a6b51cc75920d8f810c1c3b609c377164ac611b0b647b9c98cee560d3f0" exitCode=0 Oct 31 14:45:54.517339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.517320 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:45:54.517703 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.517681 4865 generic.go:356] "Generic (PLEG): container finished" podID="f1152975-ff11-49bf-b390-022434aae2a0" containerID="f531565e6ec6483ac0f8cb529d19764638087e6a422d055be1187f2a631bfd5e" exitCode=1 Oct 31 14:45:54.526972 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.526953 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/1.log" Oct 31 14:45:54.527042 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.526987 4865 generic.go:356] "Generic (PLEG): container finished" podID="1d447892-64c2-41a6-a155-65d0717627fd" containerID="806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103" exitCode=255 Oct 31 14:45:54.550719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.550699 4865 manager.go:324] Recovery completed Oct 31 14:45:54.557183 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.557164 4865 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" Oct 31 14:45:54.567156 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.567136 4865 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:54.568287 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.568268 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:54.568351 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.568308 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:54.568351 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.568320 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:54.570662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570644 4865 cpu_manager.go:226] "Starting" policy="none" Oct 31 14:45:54.570662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570661 4865 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 14:45:54.570807 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570681 4865 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 14:45:54.570884 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570864 4865 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 14:45:54.570951 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570886 4865 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 14:45:54.570951 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570919 4865 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 14:45:54.570951 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570928 4865 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 14:45:54.570951 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570936 4865 policy_none.go:50] "Start" Oct 31 14:45:54.570951 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570951 4865 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 14:45:54.571187 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.570965 4865 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 14:45:54.572015 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.571991 4865 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:54.573235 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.573219 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:54.573332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.573254 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:54.573332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.573275 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:54.583142 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.583123 4865 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 14:45:54.583218 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.583144 4865 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 14:45:54.583218 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.583156 4865 policy_none.go:44] "Start" Oct 31 14:45:54.587602 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.587579 4865 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 14:45:54.588421 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.588401 4865 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 14:45:54.590342 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590331 4865 manager.go:341] "Starting Device Plugin manager" Oct 31 14:45:54.590405 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.590359 4865 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 14:45:54.590405 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590371 4865 server.go:87] "Starting device plugin registration server" Oct 31 14:45:54.590687 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590675 4865 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 14:45:54.590719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590690 4865 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 14:45:54.590813 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590796 4865 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 14:45:54.590972 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590933 4865 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 14:45:54.591071 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590979 4865 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 14:45:54.591071 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.590986 4865 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 14:45:54.591181 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.591037 4865 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T14:45:54.590967646Z","UUID":"efadb9e0-41e8-404e-bf63-a716c0ac3830","Handler":null,"Name":"","Endpoint":""} Oct 31 14:45:54.592231 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.592208 4865 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 14:45:54.592306 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.592255 4865 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-60.us-west-1.compute.internal\" not found" Oct 31 14:45:54.592494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.592463 4865 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 14:45:54.592542 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.592502 4865 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 14:45:54.652998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.652972 4865 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 14:45:54.657669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.657652 4865 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:54.691777 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.691760 4865 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:54.692757 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.692741 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:54.692824 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.692768 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:54.692824 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.692779 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:54.692904 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.692825 4865 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.708958 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.708938 4865 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:54.709061 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.708990 4865 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:54.709061 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.709034 4865 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:54.789303 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789224 4865 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal"] Oct 31 14:45:54.789861 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789812 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"3e9c00f3a8edeac7e2ac58fb07ea319e","Type":"ContainerStarted","Data":"1a6bb78cdb206eb4f92bf29f97e5f6878b3ec98d806d0764cdf63adc172f1156"} Oct 31 14:45:54.789971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789865 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"3e9c00f3a8edeac7e2ac58fb07ea319e","Type":"ContainerDied","Data":"aa8b5a6b51cc75920d8f810c1c3b609c377164ac611b0b647b9c98cee560d3f0"} Oct 31 14:45:54.789971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789881 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"3e9c00f3a8edeac7e2ac58fb07ea319e","Type":"ContainerStarted","Data":"394c881a3914c2e94af9cec325526b945ddb46d29db395a1101c6da7f0b51fb1"} Oct 31 14:45:54.789971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789939 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"71acc277a69aa4c1becf340e4de5eca3","Type":"ContainerStarted","Data":"6c8c3ba70f4b358924bef6cda2d71ca15ddad854159515095a7dd315ec9000e9"} Oct 31 14:45:54.789971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789955 4865 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.790101 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789975 4865 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.790101 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.789956 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" event={"ID":"71acc277a69aa4c1becf340e4de5eca3","Type":"ContainerStarted","Data":"aff5feef2559574b923cefd5951dd0f31247fbd43a8fd3ab717b4b6d91d56382"} Oct 31 14:45:54.810224 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.810204 4865 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.810291 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.810259 4865 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.810823 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.810808 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeNotReady" Oct 31 14:45:54.810902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.810843 4865 setters.go:546] "Node became not ready" node="ip-10-0-131-60.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T14:45:54Z","lastTransitionTime":"2026-10-31T14:45:54Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 14:45:54.829729 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.829704 4865 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-60.us-west-1.compute.internal" is owned by different node (oldNodeID="100d4bb9-44c7-4df4-b377-92bb9e3124ab", newNodeID=""), cleaning up... Oct 31 14:45:54.868223 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.868199 4865 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:54.868310 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.868243 4865 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.869183 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.869169 4865 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:54.869246 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:54.869208 4865 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.955397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.955376 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.955510 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.955405 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:54.955510 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:54.955422 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71acc277a69aa4c1becf340e4de5eca3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"71acc277a69aa4c1becf340e4de5eca3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.056014 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.055987 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.056096 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.056007 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.056096 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.056032 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.056096 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.056059 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71acc277a69aa4c1becf340e4de5eca3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"71acc277a69aa4c1becf340e4de5eca3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.056096 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.056084 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9c00f3a8edeac7e2ac58fb07ea319e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"3e9c00f3a8edeac7e2ac58fb07ea319e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.056210 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.056098 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/71acc277a69aa4c1becf340e4de5eca3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" (UID: \"71acc277a69aa4c1becf340e4de5eca3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.440666 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.440577 4865 apiserver.go:52] "Watching apiserver" Oct 31 14:45:55.545598 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.545578 4865 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.545859 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.545844 4865 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.598179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.598149 4865 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 14:45:55.598796 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.598774 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-csvn8","kube-system/konnectivity-agent-2bgb4","openshift-dns/node-resolver-dfq2r","openshift-image-registry/image-registry-6ddb7677d5-drb9t","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj","openshift-console-operator/console-operator-7cd56b8dbc-gdnkp","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m","openshift-dns/dns-default-sptz6","openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal","openshift-image-registry/node-ca-zgrl5","openshift-multus/network-metrics-daemon-6jlkx","openshift-ovn-kubernetes/ovnkube-node-4knfg","kube-system/kas-connection-checker-685895c7c9-gq2qd","openshift-multus/multus-l8qs8","kube-system/kas-connection-checker-685895c7c9-b4frh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k","openshift-insights/insights-runtime-extractor-clmnr","openshift-network-console/networking-console-plugin-86c478bb88-7fv54","kube-system/global-pull-secret-syncer-h78q8","openshift-ingress/router-default-699cdb64d9-r57rx","openshift-ingress-canary/ingress-canary-vh7rz","openshift-insights/insights-operator-7c4986d449-n4mj8","openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2","kube-system/kas-connection-checker-685895c7c9-5p7xg","kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-n64fd","openshift-network-diagnostics/network-check-target-jrfkr","openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl","openshift-network-operator/iptables-alerter-g7z6v","openshift-image-registry/image-registry-678477bddd-n4c79"] Oct 31 14:45:55.600773 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.600722 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:55.601335 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.601316 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:55.602102 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.602079 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.602820 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.602799 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.603199 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.603182 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:55.603535 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.603520 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:55.605533 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.605515 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.608630 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.608600 4865 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:55.609306 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.609290 4865 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:55.611672 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.611657 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.612688 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.612672 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.648674 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.648646 4865 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.648674 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.648673 4865 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal" Oct 31 14:45:55.654937 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.654913 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 14:45:55.655144 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.655109 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gsm2m\"" Oct 31 14:45:55.655144 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.655126 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dm4bk\"" Oct 31 14:45:55.655278 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.654923 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 14:45:55.655522 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.655505 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sffft\"" Oct 31 14:45:55.655659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.655545 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.655659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.655627 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.655868 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.655835 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.655987 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.655973 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.656204 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.656183 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 14:45:55.656299 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.656283 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.656537 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.656520 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whbtk\"" Oct 31 14:45:55.656633 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.656538 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 14:45:55.656711 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.656690 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gm7jd\"" Oct 31 14:45:55.656784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.656749 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mg2kx\"" Oct 31 14:45:55.656784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.656772 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xknrx\"" Oct 31 14:45:55.657098 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.657063 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 14:45:55.657638 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.657618 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.657971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.657951 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 14:45:55.659383 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.659361 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 14:45:55.660068 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660051 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmnq\" (UniqueName: \"kubernetes.io/projected/e3ce5b4f-032b-4791-bc7d-0e5916aba070-kube-api-access-fjmnq\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.660143 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660079 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-bin\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.660143 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660097 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ttx\" (UniqueName: \"kubernetes.io/projected/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-kube-api-access-27ttx\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.660330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660310 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2f6\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.660391 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660342 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.660391 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660368 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477-kube-api-access-bqplf\") pod \"volume-data-source-validator-58c9f5f46d-4lmnj\" (UID: \"a8319738-e6b9-47aa-9ceb-5f0bf4d5f477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" Oct 31 14:45:55.660507 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660414 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cwp\" (UniqueName: \"kubernetes.io/projected/45be3fe3-424d-4775-8203-ec94e0853bc1-kube-api-access-v5cwp\") pod \"kas-connection-checker-685895c7c9-b4frh\" (UID: \"45be3fe3-424d-4775-8203-ec94e0853bc1\") " pod="kube-system/kas-connection-checker-685895c7c9-b4frh" Oct 31 14:45:55.660507 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660489 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqw8\" (UniqueName: \"kubernetes.io/projected/8bffb79a-a3c4-4fc8-b742-148342ad6c35-kube-api-access-4nqw8\") pod \"kas-connection-checker-685895c7c9-gq2qd\" (UID: \"8bffb79a-a3c4-4fc8-b742-148342ad6c35\") " pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" Oct 31 14:45:55.660599 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660519 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-agent-certs\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:55.660599 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660571 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.660686 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660604 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.660686 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660621 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b0ed9d-d90a-497a-8863-7605a3c89c75-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:55.660686 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660672 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-system-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.660826 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660690 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-kubelet\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.660826 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660743 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-multus-certs\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.660826 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660772 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.660909 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660862 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-sys\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.660909 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660888 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-log-socket\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.660969 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660916 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-netd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.660969 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660959 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b8335-545a-4af4-a81b-8e4efee557e8-tmp-dir\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:55.661022 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660981 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-config-volume\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.661022 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660996 4865 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 14:45:55.661100 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661025 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b8335-545a-4af4-a81b-8e4efee557e8-tmp-dir\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:55.661100 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.660996 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-etc-kubernetes\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.661100 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661072 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-tmp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.661100 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661091 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-etc-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.661254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661108 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-env-overrides\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.661254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661141 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-tmp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.661254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661164 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfksq\" (UniqueName: \"kubernetes.io/projected/4f7891ca-5583-456c-b2e4-e65723e27b16-kube-api-access-mfksq\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:55.661254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661197 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-hostroot\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.661254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661218 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-data-volume\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.661254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661236 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xcg\" (UniqueName: \"kubernetes.io/projected/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-api-access-s8xcg\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.661454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661255 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.661454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661273 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-data-volume\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.661454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661279 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.661454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661324 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-var-lib-kubelet\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.661454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661366 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245fz\" (UniqueName: \"kubernetes.io/projected/d1a946f2-c459-4f23-8f9a-91935eeb49b0-kube-api-access-245fz\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:55.661454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661427 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.661454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661445 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.661727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661462 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.661727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661490 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-conf\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.661727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661515 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tdm\" (UniqueName: \"kubernetes.io/projected/959b8335-545a-4af4-a81b-8e4efee557e8-kube-api-access-d8tdm\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:55.661727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661545 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a946f2-c459-4f23-8f9a-91935eeb49b0-cert\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:55.661727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661558 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.661727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661611 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.661727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661652 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c9bb56-e9d1-4328-a6a8-db47dba6268e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661732 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-etc-tuned\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661753 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfxw\" (UniqueName: \"kubernetes.io/projected/14236efa-49b0-452b-8136-157cdb6390cf-kube-api-access-rkfxw\") pod \"kas-connection-checker-685895c7c9-5p7xg\" (UID: \"14236efa-49b0-452b-8136-157cdb6390cf\") " pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661769 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-tmp\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661786 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661801 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfktw\" (UniqueName: \"kubernetes.io/projected/0bafcc61-160a-4f2a-9081-29bd47254408-kube-api-access-jfktw\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661805 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58e34a13-1454-4b53-9667-23944e457c8d-etc-tuned\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661816 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-snapshots\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661833 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661869 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv59l\" (UniqueName: \"kubernetes.io/projected/f3b0ed9d-d90a-497a-8863-7605a3c89c75-kube-api-access-gv59l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661878 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-tmp\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661892 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661905 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f9259fbe-036b-4629-a947-2dcd24185c78-snapshots\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.661987 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-k8s-cni-cncf-io\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.662035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662017 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-sys-fs\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662043 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-kubelet-config\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662066 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-dbus\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662106 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1152975-ff11-49bf-b390-022434aae2a0-ovn-node-metrics-cert\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662130 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b0ed9d-d90a-497a-8863-7605a3c89c75-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662148 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662163 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-konnectivity-ca\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-netns\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662314 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-systemd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662321 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f3b0ed9d-d90a-497a-8863-7605a3c89c75-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662343 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-tmp-dir\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662370 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-systemd-units\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662393 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-ovn\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662410 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-tmp-dir\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662421 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662448 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bafcc61-160a-4f2a-9081-29bd47254408-serviceca\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662492 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9259fbe-036b-4629-a947-2dcd24185c78-serving-cert\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662514 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-multus\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.662523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662531 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-multus-daemon-config\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662546 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662570 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-slash\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662601 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662630 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9259fbe-036b-4629-a947-2dcd24185c78-serving-cert\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662682 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662739 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9tt\" (UniqueName: \"kubernetes.io/projected/0a562fc8-6e09-49d5-9dcf-3188700438eb-kube-api-access-rp9tt\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662766 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-run\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662793 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttct\" (UniqueName: \"kubernetes.io/projected/f1152975-ff11-49bf-b390-022434aae2a0-kube-api-access-kttct\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662823 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-bin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662878 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-crio-socket\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662936 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/f9259fbe-036b-4629-a947-2dcd24185c78-kube-api-access-pxh9b\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.662983 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-socket-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663023 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-cni-binary-copy\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663051 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-socket-dir-parent\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663075 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-modprobe-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663102 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bafcc61-160a-4f2a-9081-29bd47254408-host\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663142 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663165 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663183 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8bmx\" (UniqueName: \"kubernetes.io/projected/788d00c1-792d-47ea-bcd3-cfee09779724-kube-api-access-n8bmx\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663204 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-default-certificate\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663229 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-script-lib\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663254 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-var-lib-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663339 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663373 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663405 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-stats-auth\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663423 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-kubelet\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663438 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/959b8335-545a-4af4-a81b-8e4efee557e8-hosts-file\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663454 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663502 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-cnibin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663532 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-netns\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663553 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysconfig\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663589 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-host\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663630 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-node-log\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663660 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-registration-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663684 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-systemd\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663715 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-lib-modules\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663746 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663771 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-config\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663788 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-service-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663842 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-conf-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.663863 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663864 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-device-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.664400 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663879 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-kubernetes\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.664400 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.663894 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.719726 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.719707 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cwp\" (UniqueName: \"kubernetes.io/projected/45be3fe3-424d-4775-8203-ec94e0853bc1-kube-api-access-v5cwp\") pod \"kas-connection-checker-685895c7c9-b4frh\" (UID: \"45be3fe3-424d-4775-8203-ec94e0853bc1\") " pod="kube-system/kas-connection-checker-685895c7c9-b4frh" Oct 31 14:45:55.720674 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.720659 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqw8\" (UniqueName: \"kubernetes.io/projected/8bffb79a-a3c4-4fc8-b742-148342ad6c35-kube-api-access-4nqw8\") pod \"kas-connection-checker-685895c7c9-gq2qd\" (UID: \"8bffb79a-a3c4-4fc8-b742-148342ad6c35\") " pod="kube-system/kas-connection-checker-685895c7c9-gq2qd" Oct 31 14:45:55.722423 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.722406 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfxw\" (UniqueName: \"kubernetes.io/projected/14236efa-49b0-452b-8136-157cdb6390cf-kube-api-access-rkfxw\") pod \"kas-connection-checker-685895c7c9-5p7xg\" (UID: \"14236efa-49b0-452b-8136-157cdb6390cf\") " pod="kube-system/kas-connection-checker-685895c7c9-5p7xg" Oct 31 14:45:55.724030 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.724012 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.753619 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753592 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 14:45:55.753619 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753620 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.753619 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753621 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.753895 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753764 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.753895 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753784 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 14:45:55.753895 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753834 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 14:45:55.754159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753949 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.754159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.753960 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-service-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.754159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754051 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 14:45:55.754159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754100 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 14:45:55.754159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754148 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.754159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754159 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754187 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754275 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754279 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754280 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754306 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754408 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754457 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754630 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754698 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754705 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754709 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754710 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754752 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.754854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754824 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.754966 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755101 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755140 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755144 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755152 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755165 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755171 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755242 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755249 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755317 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755350 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755363 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755420 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755426 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755709 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755709 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755738 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755744 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755758 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:55.755801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755797 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755836 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755885 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755979 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.755998 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756162 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756533 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756565 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756628 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xcg\" (UniqueName: \"kubernetes.io/projected/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-api-access-s8xcg\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756646 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756729 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 14:45:55.756906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756810 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" Oct 31 14:45:55.757384 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756972 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" Oct 31 14:45:55.757384 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756976 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 14:45:55.757384 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.756973 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 14:45:55.757384 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.757106 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 14:45:55.758121 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.758107 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 14:45:55.758206 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.758192 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 14:45:55.758403 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.758386 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ttx\" (UniqueName: \"kubernetes.io/projected/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-kube-api-access-27ttx\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.758497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.758442 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmnq\" (UniqueName: \"kubernetes.io/projected/e3ce5b4f-032b-4791-bc7d-0e5916aba070-kube-api-access-fjmnq\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.761343 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761322 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.761408 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761394 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f3b0ed9d-d90a-497a-8863-7605a3c89c75-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:55.761518 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761502 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-config-volume\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.761657 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761637 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-env-overrides\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.761858 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761802 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfksq\" (UniqueName: \"kubernetes.io/projected/4f7891ca-5583-456c-b2e4-e65723e27b16-kube-api-access-mfksq\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:55.761956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761868 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqplf\" (UniqueName: \"kubernetes.io/projected/a8319738-e6b9-47aa-9ceb-5f0bf4d5f477-kube-api-access-bqplf\") pod \"volume-data-source-validator-58c9f5f46d-4lmnj\" (UID: \"a8319738-e6b9-47aa-9ceb-5f0bf4d5f477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4lmnj" Oct 31 14:45:55.761956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761883 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2f6\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.761956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761869 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1a946f2-c459-4f23-8f9a-91935eeb49b0-cert\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:55.761956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761915 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83c9bb56-e9d1-4328-a6a8-db47dba6268e-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:55.762197 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761963 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.762197 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.761999 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfktw\" (UniqueName: \"kubernetes.io/projected/0bafcc61-160a-4f2a-9081-29bd47254408-kube-api-access-jfktw\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:55.762197 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762130 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" Oct 31 14:45:55.762357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762203 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv59l\" (UniqueName: \"kubernetes.io/projected/f3b0ed9d-d90a-497a-8863-7605a3c89c75-kube-api-access-gv59l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m\" (UID: \"f3b0ed9d-d90a-497a-8863-7605a3c89c75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" Oct 31 14:45:55.762357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762271 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1152975-ff11-49bf-b390-022434aae2a0-ovn-node-metrics-cert\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.762460 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762409 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-konnectivity-ca\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:55.762585 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762568 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8bmx\" (UniqueName: \"kubernetes.io/projected/788d00c1-792d-47ea-bcd3-cfee09779724-kube-api-access-n8bmx\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.762677 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762665 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0bafcc61-160a-4f2a-9081-29bd47254408-serviceca\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:55.762742 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762724 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:55.762803 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762778 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-multus-daemon-config\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.762858 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.762813 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kjc\" (UniqueName: \"kubernetes.io/projected/07979459-61ec-49fc-9696-2c985d7ad790-kube-api-access-75kjc\") pod \"network-check-target-jrfkr\" (UID: \"07979459-61ec-49fc-9696-2c985d7ad790\") " pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:45:55.763105 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763090 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tdm\" (UniqueName: \"kubernetes.io/projected/959b8335-545a-4af4-a81b-8e4efee557e8-kube-api-access-d8tdm\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:55.763168 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763155 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/788d00c1-792d-47ea-bcd3-cfee09779724-cni-binary-copy\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.763628 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763614 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245fz\" (UniqueName: \"kubernetes.io/projected/d1a946f2-c459-4f23-8f9a-91935eeb49b0-kube-api-access-245fz\") pod \"ingress-canary-vh7rz\" (UID: \"d1a946f2-c459-4f23-8f9a-91935eeb49b0\") " pod="openshift-ingress-canary/ingress-canary-vh7rz" Oct 31 14:45:55.763799 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763778 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-stats-auth\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.763891 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763826 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-script-lib\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.763952 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763894 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-89zxn\"" Oct 31 14:45:55.764018 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763954 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-default-certificate\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.764018 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.763989 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1152975-ff11-49bf-b390-022434aae2a0-ovnkube-config\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764208 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-system-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764206 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttct\" (UniqueName: \"kubernetes.io/projected/f1152975-ff11-49bf-b390-022434aae2a0-kube-api-access-kttct\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764223 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-kubelet\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764248 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9tt\" (UniqueName: \"kubernetes.io/projected/0a562fc8-6e09-49d5-9dcf-3188700438eb-kube-api-access-rp9tt\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764251 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-multus-certs\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764264 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-system-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764276 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764295 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-kubelet\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764297 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-sys\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.764316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764312 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-multus-certs\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764348 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-sys\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.764354 4865 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764404 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72hw\" (UniqueName: \"kubernetes.io/projected/58930f87-6f50-486e-9ab6-b58f7aee05c2-kube-api-access-g72hw\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764400 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxh9b\" (UniqueName: \"kubernetes.io/projected/f9259fbe-036b-4629-a947-2dcd24185c78-kube-api-access-pxh9b\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.764423 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.264408994 +0000 UTC m=+1.883960422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764505 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-log-socket\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764525 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-netd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764542 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-etc-kubernetes\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764562 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764588 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2npb\" (UniqueName: \"kubernetes.io/projected/1d447892-64c2-41a6-a155-65d0717627fd-kube-api-access-t2npb\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764594 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-log-socket\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764600 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-netd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764608 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-etc-kubernetes\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764612 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-etc-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764637 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-etc-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764643 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764676 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-hostroot\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764701 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764735 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764758 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-var-lib-kubelet\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.764784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764782 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764801 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764826 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-var-lib-kubelet\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764806 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-conf\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764875 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.764896 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.264878105 +0000 UTC m=+1.884429514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764908 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764932 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d447892-64c2-41a6-a155-65d0717627fd-serving-cert\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764977 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-system-cni-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764988 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-conf\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765005 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765036 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765057 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d447892-64c2-41a6-a155-65d0717627fd-serving-cert\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765058 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.765095 4865 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765101 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765106 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765129 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdlz\" (UniqueName: \"kubernetes.io/projected/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-kube-api-access-jgdlz\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.765148 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.265131991 +0000 UTC m=+1.884683381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.764760 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-hostroot\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765180 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-k8s-cni-cncf-io\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765201 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysctl-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765230 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-k8s-cni-cncf-io\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765237 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-sys-fs\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765272 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58930f87-6f50-486e-9ab6-b58f7aee05c2-serving-cert\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765280 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-sys-fs\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765310 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765348 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-kubelet-config\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765371 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-dbus\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765396 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58930f87-6f50-486e-9ab6-b58f7aee05c2-serving-cert\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765404 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58930f87-6f50-486e-9ab6-b58f7aee05c2-config\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765412 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-kubelet-config\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765428 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-netns\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765443 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eefaca9-ad88-44ac-be6b-363497696b72-dbus\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765454 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-systemd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765487 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-netns\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765502 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-systemd-units\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765513 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-systemd\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765530 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-ovn\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765558 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-systemd-units\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765556 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765559 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-multus\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765593 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-ovn\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765599 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-multus\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765602 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58930f87-6f50-486e-9ab6-b58f7aee05c2-config\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765605 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765634 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-slash\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765657 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765672 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765687 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-run\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765706 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765748 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-slash\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765747 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.765731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765763 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-run\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765749 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-cni-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765767 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765801 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-bin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765817 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-crio-socket\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765834 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d91b481-174f-488b-97d8-d9a1c716a71d-host-slash\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765844 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-var-lib-cni-bin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765849 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-socket-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765863 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-crio-socket\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765892 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-socket-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765903 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765933 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cff\" (UniqueName: \"kubernetes.io/projected/40101b79-243a-4844-be24-6ecd1b07f67a-kube-api-access-c8cff\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765960 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-socket-dir-parent\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.765984 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-modprobe-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766007 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bafcc61-160a-4f2a-9081-29bd47254408-host\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766032 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766042 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-socket-dir-parent\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766062 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d91b481-174f-488b-97d8-d9a1c716a71d-iptables-alerter-script\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766078 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0bafcc61-160a-4f2a-9081-29bd47254408-host\") pod \"node-ca-zgrl5\" (UID: \"0bafcc61-160a-4f2a-9081-29bd47254408\") " pod="openshift-image-registry/node-ca-zgrl5" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766086 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-var-lib-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766088 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-modprobe-d\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766113 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.766137 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766144 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-config\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766194 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.766149 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766220 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766137 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-var-lib-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766157 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766250 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.766273 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.266258454 +0000 UTC m=+1.885809848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766293 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-config\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766303 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-kubelet\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766322 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/959b8335-545a-4af4-a81b-8e4efee557e8-hosts-file\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.766336 4865 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766343 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766369 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-cnibin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766380 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-kubelet\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.766390 4865 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766338 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d91b481-174f-488b-97d8-d9a1c716a71d-iptables-alerter-script\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766395 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/959b8335-545a-4af4-a81b-8e4efee557e8-hosts-file\") pod \"node-resolver-dfq2r\" (UID: \"959b8335-545a-4af4-a81b-8e4efee557e8\") " pod="openshift-dns/node-resolver-dfq2r" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766393 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-netns\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.766411 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.266397023 +0000 UTC m=+1.885948412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766421 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-host-run-netns\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766437 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-cnibin\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.766451 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.26644337 +0000 UTC m=+1.885994758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766483 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysconfig\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766509 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-host\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766517 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-sysconfig\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766533 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-node-log\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766562 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-registration-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766575 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-systemd\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766581 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-host\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766590 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blljj\" (UniqueName: \"kubernetes.io/projected/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-kube-api-access-blljj\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766605 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-lib-modules\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766616 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-registration-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766621 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cnibin\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766636 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwjg\" (UniqueName: \"kubernetes.io/projected/7d91b481-174f-488b-97d8-d9a1c716a71d-kube-api-access-8qwjg\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766639 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-node-log\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766652 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766670 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766686 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-conf-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766701 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-lib-modules\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766726 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766701 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-device-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766743 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-systemd\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766770 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-kubernetes\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766772 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a562fc8-6e09-49d5-9dcf-3188700438eb-device-dir\") pod \"aws-ebs-csi-driver-node-g8f5k\" (UID: \"0a562fc8-6e09-49d5-9dcf-3188700438eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8f5k" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766774 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/788d00c1-792d-47ea-bcd3-cfee09779724-multus-conf-dir\") pod \"multus-l8qs8\" (UID: \"788d00c1-792d-47ea-bcd3-cfee09779724\") " pod="openshift-multus/multus-l8qs8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766789 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g799v\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766807 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766821 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e34a13-1454-4b53-9667-23944e457c8d-etc-kubernetes\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766830 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-bin\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766848 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766868 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766890 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766899 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-host-cni-bin\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766917 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9tp\" (UniqueName: \"kubernetes.io/projected/58e34a13-1454-4b53-9667-23944e457c8d-kube-api-access-pg9tp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766920 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1152975-ff11-49bf-b390-022434aae2a0-run-openvswitch\") pod \"ovnkube-node-4knfg\" (UID: \"f1152975-ff11-49bf-b390-022434aae2a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766923 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766946 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdr68\" (UniqueName: \"kubernetes.io/projected/810a09fc-7671-46ff-a0d9-222437cb00a1-kube-api-access-vdr68\") pod \"network-check-source-6f78d59f99-2p6pl\" (UID: \"810a09fc-7671-46ff-a0d9-222437cb00a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766968 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.766985 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.767785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.767163 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.770035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.768419 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" Oct 31 14:45:55.770035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.768434 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 14:45:55.770035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.768486 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fzc6l\"" Oct 31 14:45:55.770424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.770410 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" Oct 31 14:45:55.770424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.770420 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" Oct 31 14:45:55.771361 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.771346 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 14:45:55.771361 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.771357 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t2tz4\"" Oct 31 14:45:55.771523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.771404 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tgszw\"" Oct 31 14:45:55.771523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.771452 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4-agent-certs\") pod \"konnectivity-agent-2bgb4\" (UID: \"6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4\") " pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:45:55.772533 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.772520 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 14:45:55.772591 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.772549 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 14:45:55.772591 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.772575 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" Oct 31 14:45:55.772655 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.772596 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" Oct 31 14:45:55.772655 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.772604 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" Oct 31 14:45:55.773089 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.773075 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.773339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.773324 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 14:45:55.773339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.773326 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" Oct 31 14:45:55.773424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.773363 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" Oct 31 14:45:55.773424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.773368 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" Oct 31 14:45:55.773424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.773397 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" Oct 31 14:45:55.774426 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.774412 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" Oct 31 14:45:55.774731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.774718 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" Oct 31 14:45:55.775783 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.775770 4865 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:55.775844 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.775822 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.275806275 +0000 UTC m=+1.895357669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:55.775898 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.775867 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.776972 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.776960 4865 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:55.777029 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.777005 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.276991617 +0000 UTC m=+1.896543011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:55.782708 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.782693 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eefaca9-ad88-44ac-be6b-363497696b72-original-pull-secret\") pod \"global-pull-secret-syncer-h78q8\" (UID: \"8eefaca9-ad88-44ac-be6b-363497696b72\") " pod="kube-system/global-pull-secret-syncer-h78q8" Oct 31 14:45:55.805971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.805950 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 14:45:55.811852 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.808799 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 14:45:55.811852 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.809990 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 14:45:55.812679 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.812656 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:55.812848 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.812827 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9259fbe-036b-4629-a947-2dcd24185c78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-n4mj8\" (UID: \"f9259fbe-036b-4629-a947-2dcd24185c78\") " pod="openshift-insights/insights-operator-7c4986d449-n4mj8" Oct 31 14:45:55.816267 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.816249 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d447892-64c2-41a6-a155-65d0717627fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.817363 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.817346 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.822381 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.822361 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72hw\" (UniqueName: \"kubernetes.io/projected/58930f87-6f50-486e-9ab6-b58f7aee05c2-kube-api-access-g72hw\") pod \"service-ca-operator-544d749c8c-hmvx2\" (UID: \"58930f87-6f50-486e-9ab6-b58f7aee05c2\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" Oct 31 14:45:55.822547 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.822528 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2npb\" (UniqueName: \"kubernetes.io/projected/1d447892-64c2-41a6-a155-65d0717627fd-kube-api-access-t2npb\") pod \"console-operator-7cd56b8dbc-gdnkp\" (UID: \"1d447892-64c2-41a6-a155-65d0717627fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:45:55.822607 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.822555 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdlz\" (UniqueName: \"kubernetes.io/projected/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-kube-api-access-jgdlz\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.823256 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.823240 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.826263 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.826236 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwjg\" (UniqueName: \"kubernetes.io/projected/7d91b481-174f-488b-97d8-d9a1c716a71d-kube-api-access-8qwjg\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:55.826336 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.826265 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blljj\" (UniqueName: \"kubernetes.io/projected/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-kube-api-access-blljj\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:55.826336 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.826320 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9tp\" (UniqueName: \"kubernetes.io/projected/58e34a13-1454-4b53-9667-23944e457c8d-kube-api-access-pg9tp\") pod \"tuned-csvn8\" (UID: \"58e34a13-1454-4b53-9667-23944e457c8d\") " pod="openshift-cluster-node-tuning-operator/tuned-csvn8" Oct 31 14:45:55.826432 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.826342 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g799v\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.826545 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.826512 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cff\" (UniqueName: \"kubernetes.io/projected/40101b79-243a-4844-be24-6ecd1b07f67a-kube-api-access-c8cff\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:55.826871 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.826855 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdr68\" (UniqueName: \"kubernetes.io/projected/810a09fc-7671-46ff-a0d9-222437cb00a1-kube-api-access-vdr68\") pod \"network-check-source-6f78d59f99-2p6pl\" (UID: \"810a09fc-7671-46ff-a0d9-222437cb00a1\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-2p6pl" Oct 31 14:45:55.867778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867753 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.867866 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867790 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d91b481-174f-488b-97d8-d9a1c716a71d-host-slash\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:55.867866 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867814 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:55.867866 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867860 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867869 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d91b481-174f-488b-97d8-d9a1c716a71d-host-slash\") pod \"iptables-alerter-g7z6v\" (UID: \"7d91b481-174f-488b-97d8-d9a1c716a71d\") " pod="openshift-network-operator/iptables-alerter-g7z6v" Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867869 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867917 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cnibin\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.867926 4865 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.867954 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.867967 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.367955617 +0000 UTC m=+1.987507027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : secret "metrics-daemon-secret" not found Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.868004 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-cnibin\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.868023 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.868025 4865 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:55.868044 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.868035 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:55.868386 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.868095 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.368079752 +0000 UTC m=+1.987631156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:55.868386 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.868114 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-system-cni-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.868386 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:55.868158 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:56.368146289 +0000 UTC m=+1.987697681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:55.868386 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.868168 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/befbecfb-5e0b-4d8b-8e13-54c7394ba5bc-system-cni-dir\") pod \"multus-additional-cni-plugins-n64fd\" (UID: \"befbecfb-5e0b-4d8b-8e13-54c7394ba5bc\") " pod="openshift-multus/multus-additional-cni-plugins-n64fd" Oct 31 14:45:55.900754 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:55.900738 4865 scope.go:122] "RemoveContainer" containerID="806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103" Oct 31 14:45:56.271228 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.271141 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:56.271228 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.271187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.271232 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271290 4865 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271343 4865 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271365 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271380 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271360 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.271340535 +0000 UTC m=+2.890891941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271402 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.271393614 +0000 UTC m=+2.890945003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.271290 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271412 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.271406737 +0000 UTC m=+2.890958126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:56.271455 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271450 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.271425082 +0000 UTC m=+2.890976475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:56.271921 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.271523 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:56.271921 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.271557 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:56.271921 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271641 4865 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:56.271921 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271679 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.271666907 +0000 UTC m=+2.891218299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:56.271921 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271707 4865 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:56.271921 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.271755 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.27174267 +0000 UTC m=+2.891294076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:56.372059 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.372026 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:56.372211 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.372095 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:56.372211 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.372126 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:56.372211 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372176 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:56.372211 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372197 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:56.372338 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.372213 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:56.372338 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372248 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.372232478 +0000 UTC m=+2.991783890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:56.372338 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372282 4865 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:56.372338 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372294 4865 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:56.372338 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.372309 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:56.372338 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372244 4865 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:56.372338 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372335 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.372318041 +0000 UTC m=+2.991869432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:56.372563 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372360 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.372353273 +0000 UTC m=+2.991904662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:56.372563 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372363 4865 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:56.372563 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372374 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.37236739 +0000 UTC m=+2.991918779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:56.372563 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.372387 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.372381592 +0000 UTC m=+2.991932981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : secret "metrics-daemon-secret" not found Oct 31 14:45:56.550296 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.550232 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 14:45:56.550710 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.550696 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/1.log" Oct 31 14:45:56.550747 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.550728 4865 generic.go:356] "Generic (PLEG): container finished" podID="1d447892-64c2-41a6-a155-65d0717627fd" containerID="36d5bd3dac401d7e2604ff47b1bc02d4477391a7f13cf95909e7b3f4e69932a0" exitCode=255 Oct 31 14:45:56.550831 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.550807 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" event={"ID":"1d447892-64c2-41a6-a155-65d0717627fd","Type":"ContainerDied","Data":"36d5bd3dac401d7e2604ff47b1bc02d4477391a7f13cf95909e7b3f4e69932a0"} Oct 31 14:45:56.550959 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.550864 4865 scope.go:122] "RemoveContainer" containerID="806cbfde9e77b8c162f1d7ede4d3214d7d32fd2f4cfa3b3151421c28a4255103" Oct 31 14:45:56.551257 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.551240 4865 scope.go:122] "RemoveContainer" containerID="36d5bd3dac401d7e2604ff47b1bc02d4477391a7f13cf95909e7b3f4e69932a0" Oct 31 14:45:56.551447 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:56.551429 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gdnkp_openshift-console-operator(1d447892-64c2-41a6-a155-65d0717627fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" podUID="1d447892-64c2-41a6-a155-65d0717627fd" Oct 31 14:45:56.603260 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:56.603239 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:57.281684 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.281654 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.281696 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.281717 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.281775 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281796 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281812 4865 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.281804 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281867 4865 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281817 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:57.281893 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281876 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.281857942 +0000 UTC m=+4.901409335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281896 4865 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281919 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.28190418 +0000 UTC m=+4.901455571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281940 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.281929402 +0000 UTC m=+4.901480805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281957 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.281951075 +0000 UTC m=+4.901502464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.281968 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.281963494 +0000 UTC m=+4.901514883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.281994 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.282051 4865 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:57.282297 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.282083 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.28207475 +0000 UTC m=+4.901626161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:57.382995 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.382966 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:57.383178 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.383036 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:57.383178 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.383065 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:57.383178 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383125 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:57.383178 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383132 4865 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:57.383178 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383140 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:57.383178 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383142 4865 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383191 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.383172891 +0000 UTC m=+5.002724280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383209 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.383200345 +0000 UTC m=+5.002751739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383225 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.383215937 +0000 UTC m=+5.002767325 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.383303 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383393 4865 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.383394 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383419 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.383409119 +0000 UTC m=+5.002960509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:57.383490 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383464 4865 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:57.383758 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:57.383533 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.383521607 +0000 UTC m=+5.003072998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : secret "metrics-daemon-secret" not found Oct 31 14:45:57.555406 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:57.555333 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 14:45:58.374009 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:58.373984 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:45:58.507612 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:58.507588 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dfq2r_959b8335-545a-4af4-a81b-8e4efee557e8/dns-node-resolver/0.log" Oct 31 14:45:58.871503 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:58.871451 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:59.301200 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.301158 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:45:59.301383 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.301225 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:45:59.301383 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.301265 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:45:59.301383 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301316 4865 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 14:45:59.301383 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.301338 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:59.301383 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301375 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls podName:4f7891ca-5583-456c-b2e4-e65723e27b16 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.301360151 +0000 UTC m=+8.920911541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mdg5f" (UID: "4f7891ca-5583-456c-b2e4-e65723e27b16") : secret "samples-operator-tls" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.301394 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.301437 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301398 4865 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301492 4865 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301317 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301529 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301398 4865 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301457 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.301447172 +0000 UTC m=+8.920998585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : configmap references non-existent config key: service-ca.crt Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301594 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.301587392 +0000 UTC m=+8.921138781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301604 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs podName:e3ce5b4f-032b-4791-bc7d-0e5916aba070 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.301598815 +0000 UTC m=+8.921150203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs") pod "router-default-699cdb64d9-r57rx" (UID: "e3ce5b4f-032b-4791-bc7d-0e5916aba070") : secret "router-metrics-certs-default" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301616 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.301609707 +0000 UTC m=+8.921161095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:45:59.301652 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.301624 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert podName:83c9bb56-e9d1-4328-a6a8-db47dba6268e nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.301619597 +0000 UTC m=+8.921170986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7fv54" (UID: "83c9bb56-e9d1-4328-a6a8-db47dba6268e") : secret "networking-console-plugin-cert" not found Oct 31 14:45:59.399505 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.399458 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:59.402676 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.402639 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:45:59.402805 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.402718 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:45:59.402805 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402765 4865 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:59.402805 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.402792 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402821 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.402809351 +0000 UTC m=+9.022360741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : secret "metrics-daemon-secret" not found Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.402840 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402850 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402867 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402885 4865 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.402910 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402940 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.402925308 +0000 UTC m=+9.022476698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402940 4865 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402966 4865 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:59.402968 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402971 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.40295932 +0000 UTC m=+9.022510714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:45:59.403291 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.402989 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls podName:b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.40298243 +0000 UTC m=+9.022533818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls") pod "dns-default-sptz6" (UID: "b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8") : secret "dns-default-metrics-tls" not found Oct 31 14:45:59.403291 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:45:59.403012 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.403001398 +0000 UTC m=+9.022552786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:59.537154 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.537123 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:59.675719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.675616 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:45:59.707990 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:45:59.707932 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zgrl5_0bafcc61-160a-4f2a-9081-29bd47254408/node-ca/0.log" Oct 31 14:46:00.375370 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:00.375342 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vh7rz_d1a946f2-c459-4f23-8f9a-91935eeb49b0/serve-healthcheck-canary/0.log" Oct 31 14:46:00.570508 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:00.570462 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:46:00.570742 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:00.570726 4865 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 14:46:01.114271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:01.114239 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m_f3b0ed9d-d90a-497a-8863-7605a3c89c75/kube-storage-version-migrator-operator/0.log" Oct 31 14:46:01.132036 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:01.132010 4865 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:46:01.132383 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:01.132366 4865 scope.go:122] "RemoveContainer" containerID="36d5bd3dac401d7e2604ff47b1bc02d4477391a7f13cf95909e7b3f4e69932a0" Oct 31 14:46:01.132606 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:01.132586 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gdnkp_openshift-console-operator(1d447892-64c2-41a6-a155-65d0717627fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" podUID="1d447892-64c2-41a6-a155-65d0717627fd" Oct 31 14:46:02.911916 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:02.911882 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:46:02.912318 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:02.912225 4865 scope.go:122] "RemoveContainer" containerID="36d5bd3dac401d7e2604ff47b1bc02d4477391a7f13cf95909e7b3f4e69932a0" Oct 31 14:46:02.912443 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:02.912418 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gdnkp_openshift-console-operator(1d447892-64c2-41a6-a155-65d0717627fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" podUID="1d447892-64c2-41a6-a155-65d0717627fd" Oct 31 14:46:03.341288 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.341253 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:03.341464 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.341296 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:46:03.341464 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.341318 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:46:03.341464 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.341373 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:03.341464 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.341394 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:46:03.341464 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.341413 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678477bddd-n4c79: secret "image-registry-tls" not found Oct 31 14:46:03.341745 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.341505 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls podName:8340f44a-41ba-4624-9bc4-a7c8873ab788 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:11.341452245 +0000 UTC m=+16.961003634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls") pod "image-registry-678477bddd-n4c79" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788") : secret "image-registry-tls" not found Oct 31 14:46:03.341745 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.341399 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:03.341745 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.341567 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:46:03.341745 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.341714 4865 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:46:03.341745 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.341746 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:46:11.341737657 +0000 UTC m=+16.961289060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : configmap "kube-rbac-proxy" not found Oct 31 14:46:03.341928 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.341774 4865 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 14:46:03.342082 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.342051 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3ce5b4f-032b-4791-bc7d-0e5916aba070-service-ca-bundle\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:03.344121 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.344100 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f7891ca-5583-456c-b2e4-e65723e27b16-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mdg5f\" (UID: \"4f7891ca-5583-456c-b2e4-e65723e27b16\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:46:03.344229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.344154 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83c9bb56-e9d1-4328-a6a8-db47dba6268e-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7fv54\" (UID: \"83c9bb56-e9d1-4328-a6a8-db47dba6268e\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:46:03.344289 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.344245 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ce5b4f-032b-4791-bc7d-0e5916aba070-metrics-certs\") pod \"router-default-699cdb64d9-r57rx\" (UID: \"e3ce5b4f-032b-4791-bc7d-0e5916aba070\") " pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:03.401258 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.401234 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" Oct 31 14:46:03.402431 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.402415 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:03.403616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.403603 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" Oct 31 14:46:03.443263 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.443223 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:46:03.443409 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.443279 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:46:03.443409 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.443343 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:46:03.443598 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443403 4865 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 14:46:03.443598 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443496 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls podName:76e4782a-cfd3-4b3e-9d37-01c836d0e0ad nodeName:}" failed. No retries permitted until 2026-10-31 14:46:11.443453239 +0000 UTC m=+17.063004632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-b8jpk" (UID: "76e4782a-cfd3-4b3e-9d37-01c836d0e0ad") : secret "cluster-monitoring-operator-tls" not found Oct 31 14:46:03.443598 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.443413 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:46:03.443598 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443519 4865 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:46:03.443598 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443566 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs podName:40101b79-243a-4844-be24-6ecd1b07f67a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:11.443553413 +0000 UTC m=+17.063104832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs") pod "network-metrics-daemon-6jlkx" (UID: "40101b79-243a-4844-be24-6ecd1b07f67a") : secret "metrics-daemon-secret" not found Oct 31 14:46:03.443598 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443576 4865 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:46:03.443598 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.443587 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:46:03.443963 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443621 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls podName:a93ffb8f-f9d5-4663-a591-d94fd7ca296f nodeName:}" failed. No retries permitted until 2026-10-31 14:46:11.443609591 +0000 UTC m=+17.063160979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clmnr" (UID: "a93ffb8f-f9d5-4663-a591-d94fd7ca296f") : secret "insights-runtime-extractor-tls" not found Oct 31 14:46:03.443963 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443663 4865 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 14:46:03.443963 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443675 4865 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ddb7677d5-drb9t: secret "image-registry-tls" not found Oct 31 14:46:03.443963 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:03.443714 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls podName:dfb9d804-55ea-4c38-ad60-6aec59c7afd4 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:11.443703389 +0000 UTC m=+17.063254800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls") pod "image-registry-6ddb7677d5-drb9t" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4") : secret "image-registry-tls" not found Oct 31 14:46:03.446576 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.446525 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8-metrics-tls\") pod \"dns-default-sptz6\" (UID: \"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8\") " pod="openshift-dns/dns-default-sptz6" Oct 31 14:46:03.712910 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.712883 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sptz6" Oct 31 14:46:03.782422 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.782397 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f"] Oct 31 14:46:03.823434 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.823413 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7fv54"] Oct 31 14:46:03.834019 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.832450 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-699cdb64d9-r57rx"] Oct 31 14:46:03.834816 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:03.834795 4865 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:46:03.836917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:46:03.836887 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ce5b4f_032b_4791_bc7d_0e5916aba070.slice/crio-2594615a7c79f892116e390f588a966dbc63f3f0b84051327aa7c78e3e8f2207 WatchSource:0}: Error finding container 2594615a7c79f892116e390f588a966dbc63f3f0b84051327aa7c78e3e8f2207: Status 404 returned error can't find the container with id 2594615a7c79f892116e390f588a966dbc63f3f0b84051327aa7c78e3e8f2207 Oct 31 14:46:04.075769 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.075748 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sptz6"] Oct 31 14:46:04.077986 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:46:04.077961 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb063fbd8_33f1_4c3d_a176_7ecb9d76d5f8.slice/crio-ebae341f84ce43af5d30296580ba80b6be70e2c1d4b9196526d654563285cd54 WatchSource:0}: Error finding container ebae341f84ce43af5d30296580ba80b6be70e2c1d4b9196526d654563285cd54: Status 404 returned error can't find the container with id ebae341f84ce43af5d30296580ba80b6be70e2c1d4b9196526d654563285cd54 Oct 31 14:46:04.197603 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.197572 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:46:04.579506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.579458 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-699cdb64d9-r57rx" event={"ID":"e3ce5b4f-032b-4791-bc7d-0e5916aba070","Type":"ContainerStarted","Data":"0e2ef11596f6bbbb1255b651e3f6cc9c0c5727240421683ce0fe0d2dd7a51c0b"} Oct 31 14:46:04.579710 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.579531 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-699cdb64d9-r57rx" event={"ID":"e3ce5b4f-032b-4791-bc7d-0e5916aba070","Type":"ContainerStarted","Data":"2594615a7c79f892116e390f588a966dbc63f3f0b84051327aa7c78e3e8f2207"} Oct 31 14:46:04.580739 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.580716 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sptz6" event={"ID":"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8","Type":"ContainerStarted","Data":"ebae341f84ce43af5d30296580ba80b6be70e2c1d4b9196526d654563285cd54"} Oct 31 14:46:04.581845 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.581825 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" event={"ID":"83c9bb56-e9d1-4328-a6a8-db47dba6268e","Type":"ContainerStarted","Data":"7a2ae2bc8299d73b47ccf501def58c20683f4f83611bb1867db3c6de107865d0"} Oct 31 14:46:04.582878 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.582861 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" event={"ID":"4f7891ca-5583-456c-b2e4-e65723e27b16","Type":"ContainerStarted","Data":"95e27d13b794818d82179d08cfd157ef511466cd2a5e4fb704936c59ded40244"} Oct 31 14:46:04.901693 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:04.901623 4865 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-60.us-west-1.compute.internal" event="NodeReady" Oct 31 14:46:05.041946 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.041527 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:05.220168 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.220098 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:05.342744 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.342712 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:05.403495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.403419 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:05.409064 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.408816 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:05.475440 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.475359 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:05.590058 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.590010 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7fv54" event={"ID":"83c9bb56-e9d1-4328-a6a8-db47dba6268e","Type":"ContainerStarted","Data":"dbc8c5b8235f7d3df480ad8053754bef8a68370bdeb79c79e3e18da728e8d7e8"} Oct 31 14:46:05.837403 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:05.837309 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:06.594367 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:06.594332 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" event={"ID":"4f7891ca-5583-456c-b2e4-e65723e27b16","Type":"ContainerStarted","Data":"5c6eb6e82c6e0d222ab56e9239fe318d50184475c92fb4cbc78eb49eb0e719a8"} Oct 31 14:46:06.595018 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:06.594391 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mdg5f" event={"ID":"4f7891ca-5583-456c-b2e4-e65723e27b16","Type":"ContainerStarted","Data":"980f98cb1a15d8cc189cde41201d45579f57aad7fdf66e92b5d4d384d5329072"} Oct 31 14:46:06.596267 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:06.596244 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sptz6" event={"ID":"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8","Type":"ContainerStarted","Data":"aacc385783c94c277f254cd31fed2ecbce6f9c75a9298f330568ef995736a07f"} Oct 31 14:46:06.596339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:06.596295 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sptz6" event={"ID":"b063fbd8-33f1-4c3d-a176-7ecb9d76d5f8","Type":"ContainerStarted","Data":"4457dc26374659ef8ff2d62803fd5f544536a5bafeead052d3433eba12259495"} Oct 31 14:46:06.596434 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:06.596419 4865 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 14:46:06.630684 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:06.630658 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:07.030323 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.030287 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:07.234860 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.234828 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:07.436292 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.436212 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:07.488869 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.488841 4865 scope.go:122] "RemoveContainer" containerID="36d5bd3dac401d7e2604ff47b1bc02d4477391a7f13cf95909e7b3f4e69932a0" Oct 31 14:46:07.600794 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.600771 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 14:46:07.601180 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.600890 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" event={"ID":"1d447892-64c2-41a6-a155-65d0717627fd","Type":"ContainerStarted","Data":"0194b5b74b47768ba14fc777cc4eee98bf26d74944ac9636c62c1f473c8cdf65"} Oct 31 14:46:07.601233 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.601216 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sptz6" Oct 31 14:46:07.601500 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.601485 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:46:07.602532 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.602504 4865 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-gdnkp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.21:8443/readyz\": dial tcp 10.132.0.21:8443: connect: connection refused" start-of-body= Oct 31 14:46:07.602618 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.602593 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" podUID="1d447892-64c2-41a6-a155-65d0717627fd" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.21:8443/readyz\": dial tcp 10.132.0.21:8443: connect: connection refused" Oct 31 14:46:07.885159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:07.885104 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:08.184773 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.184709 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4knfg" Oct 31 14:46:08.409387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.409355 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:08.409600 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.409501 4865 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 14:46:08.410810 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.410790 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-699cdb64d9-r57rx" Oct 31 14:46:08.418702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.418679 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:08.608241 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.608214 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gdnkp" Oct 31 14:46:08.637897 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.637871 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:08.941803 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:08.941733 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:09.071873 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:09.071842 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:09.234551 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:09.234526 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:09.370599 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:09.370567 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:09.512663 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:09.512596 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:09.639388 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:09.639361 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:10.034672 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:10.034639 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:10.164831 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:10.164798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:10.304809 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:10.304740 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:11.009713 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.009681 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ddb7677d5-drb9t"] Oct 31 14:46:11.010081 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:11.009950 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" podUID="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" Oct 31 14:46:11.071829 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.071805 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:11.216804 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.216774 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:11.370713 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.370664 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:11.420942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.420914 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:46:11.421098 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.420974 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:11.421452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.421436 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:46:11.423490 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.423454 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"image-registry-678477bddd-n4c79\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:11.506156 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.506090 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:11.521651 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.521624 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:46:11.521789 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.521674 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:46:11.521839 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.521817 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:46:11.521967 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.521947 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:46:11.524423 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.524392 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a93ffb8f-f9d5-4663-a591-d94fd7ca296f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clmnr\" (UID: \"a93ffb8f-f9d5-4663-a591-d94fd7ca296f\") " pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:46:11.524766 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.524745 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e4782a-cfd3-4b3e-9d37-01c836d0e0ad-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-b8jpk\" (UID: \"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:46:11.524827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.524765 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"image-registry-6ddb7677d5-drb9t\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:46:11.524922 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.524907 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40101b79-243a-4844-be24-6ecd1b07f67a-metrics-certs\") pod \"network-metrics-daemon-6jlkx\" (UID: \"40101b79-243a-4844-be24-6ecd1b07f67a\") " pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:46:11.555076 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.554896 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:11.612733 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.612704 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:46:11.618971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.618949 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:46:11.695712 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.695679 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:11.723918 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.723895 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724038 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.723928 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724038 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.723967 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724117 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724093 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724166 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724126 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724210 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724166 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724210 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724186 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724306 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724241 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\") pod \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\" (UID: \"dfb9d804-55ea-4c38-ad60-6aec59c7afd4\") " Oct 31 14:46:11.724549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724462 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:46:11.724680 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724566 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:46:11.724680 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.724642 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:46:11.726760 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.726732 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:46:11.726865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.726786 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:46:11.726865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.726802 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:46:11.726865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.726816 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:46:11.726865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.726843 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v" pod "dfb9d804-55ea-4c38-ad60-6aec59c7afd4" (UID: "dfb9d804-55ea-4c38-ad60-6aec59c7afd4"). InnerVolumeSpecName "kube-api-access-g799v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:46:11.802125 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.802065 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" Oct 31 14:46:11.803772 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.803757 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jlkx" Oct 31 14:46:11.813315 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.813293 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clmnr" Oct 31 14:46:11.825817 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825793 4865 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-installation-pull-secrets\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.825902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825823 4865 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-tls\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.825902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825843 4865 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-trusted-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.825902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825857 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g799v\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-kube-api-access-g799v\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.825902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825871 4865 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-bound-sa-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.825902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825885 4865 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-registry-certificates\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.825902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825899 4865 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-image-registry-private-configuration\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.826191 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.825915 4865 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfb9d804-55ea-4c38-ad60-6aec59c7afd4-ca-trust-extracted\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:11.938205 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:11.938150 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678477bddd-n4c79"] Oct 31 14:46:11.940705 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:46:11.940681 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8340f44a_41ba_4624_9bc4_a7c8873ab788.slice/crio-d333d1c7b6d1fa81e9b37be8b4fd00483bdc8ffc89e0fa348bc3829fc1c394c4 WatchSource:0}: Error finding container d333d1c7b6d1fa81e9b37be8b4fd00483bdc8ffc89e0fa348bc3829fc1c394c4: Status 404 returned error can't find the container with id d333d1c7b6d1fa81e9b37be8b4fd00483bdc8ffc89e0fa348bc3829fc1c394c4 Oct 31 14:46:12.176583 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.176552 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk"] Oct 31 14:46:12.179428 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:46:12.179402 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e4782a_cfd3_4b3e_9d37_01c836d0e0ad.slice/crio-d430de42b38893f9a45db02db284e661fc96d1bac77becdd69528945e0e9e907 WatchSource:0}: Error finding container d430de42b38893f9a45db02db284e661fc96d1bac77becdd69528945e0e9e907: Status 404 returned error can't find the container with id d430de42b38893f9a45db02db284e661fc96d1bac77becdd69528945e0e9e907 Oct 31 14:46:12.179557 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.179412 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jlkx"] Oct 31 14:46:12.181574 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:46:12.181558 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40101b79_243a_4844_be24_6ecd1b07f67a.slice/crio-d8b28a3e2e11034d2b6d24174fb1c1258c904dcf72a39a6e2ddefd121223cbb8 WatchSource:0}: Error finding container d8b28a3e2e11034d2b6d24174fb1c1258c904dcf72a39a6e2ddefd121223cbb8: Status 404 returned error can't find the container with id d8b28a3e2e11034d2b6d24174fb1c1258c904dcf72a39a6e2ddefd121223cbb8 Oct 31 14:46:12.227981 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.227955 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clmnr"] Oct 31 14:46:12.229780 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:46:12.229753 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93ffb8f_f9d5_4663_a591_d94fd7ca296f.slice/crio-f129ccea2741b6842645480b459325d426f0e78602ffad9a05a435ff007db8fc WatchSource:0}: Error finding container f129ccea2741b6842645480b459325d426f0e78602ffad9a05a435ff007db8fc: Status 404 returned error can't find the container with id f129ccea2741b6842645480b459325d426f0e78602ffad9a05a435ff007db8fc Oct 31 14:46:12.620064 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.620026 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clmnr" event={"ID":"a93ffb8f-f9d5-4663-a591-d94fd7ca296f","Type":"ContainerStarted","Data":"ba667b1e996fe0d87cc53a9cf6ea84c2932100cca9b5567a393bdc7f4fd796b0"} Oct 31 14:46:12.620232 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.620096 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clmnr" event={"ID":"a93ffb8f-f9d5-4663-a591-d94fd7ca296f","Type":"ContainerStarted","Data":"f129ccea2741b6842645480b459325d426f0e78602ffad9a05a435ff007db8fc"} Oct 31 14:46:12.621212 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.621185 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jlkx" event={"ID":"40101b79-243a-4844-be24-6ecd1b07f67a","Type":"ContainerStarted","Data":"d8b28a3e2e11034d2b6d24174fb1c1258c904dcf72a39a6e2ddefd121223cbb8"} Oct 31 14:46:12.622317 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.622297 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" event={"ID":"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad","Type":"ContainerStarted","Data":"d430de42b38893f9a45db02db284e661fc96d1bac77becdd69528945e0e9e907"} Oct 31 14:46:12.623716 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.623699 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddb7677d5-drb9t" Oct 31 14:46:12.623820 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.623716 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678477bddd-n4c79" event={"ID":"8340f44a-41ba-4624-9bc4-a7c8873ab788","Type":"ContainerStarted","Data":"89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07"} Oct 31 14:46:12.623820 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.623751 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678477bddd-n4c79" event={"ID":"8340f44a-41ba-4624-9bc4-a7c8873ab788","Type":"ContainerStarted","Data":"d333d1c7b6d1fa81e9b37be8b4fd00483bdc8ffc89e0fa348bc3829fc1c394c4"} Oct 31 14:46:12.624035 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.624021 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:12.866769 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.866735 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ddb7677d5-drb9t"] Oct 31 14:46:12.870314 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:12.870291 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6ddb7677d5-drb9t"] Oct 31 14:46:13.636361 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:13.636207 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clmnr" event={"ID":"a93ffb8f-f9d5-4663-a591-d94fd7ca296f","Type":"ContainerStarted","Data":"7d8678ae87998c0e2c9fad02a00a5b80554c9c5922412bbaffb55fb38dcb3d22"} Oct 31 14:46:14.025298 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:14.025266 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2bgb4" Oct 31 14:46:14.495235 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:14.495204 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb9d804-55ea-4c38-ad60-6aec59c7afd4" path="/var/lib/kubelet/pods/dfb9d804-55ea-4c38-ad60-6aec59c7afd4/volumes" Oct 31 14:46:14.658400 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:14.658363 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jlkx" event={"ID":"40101b79-243a-4844-be24-6ecd1b07f67a","Type":"ContainerStarted","Data":"159abcf078c3bbe3c3846084b7a2243453a618785cf23b102cba51f2116a5f3f"} Oct 31 14:46:14.658914 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:14.658423 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jlkx" event={"ID":"40101b79-243a-4844-be24-6ecd1b07f67a","Type":"ContainerStarted","Data":"39a1aa65cf1d5e098eeb0f0087d4257ca0a64c9665ae5cd3fde808dcc9255a24"} Oct 31 14:46:14.660493 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:14.660447 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-b8jpk" event={"ID":"76e4782a-cfd3-4b3e-9d37-01c836d0e0ad","Type":"ContainerStarted","Data":"bbeb291ef9c40cbb49fe3bf2aabd20e9c4d3f8b32d43791f4e9bfdcfd38e61e0"} Oct 31 14:46:14.894835 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:14.894756 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:15.019516 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:15.019457 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:15.436171 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:15.436132 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:15.567362 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:15.567257 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:15.666441 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:15.666406 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clmnr" event={"ID":"a93ffb8f-f9d5-4663-a591-d94fd7ca296f","Type":"ContainerStarted","Data":"c0659e07c88f307c4064abb02c1b9c64593cae8e72c8c176b5ec904f3d0d8c02"} Oct 31 14:46:15.701074 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:15.701015 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:16.488639 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:16.488599 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:16.697894 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:16.697858 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:17.488987 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:17.488953 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:46:17.606001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:17.605973 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sptz6" Oct 31 14:46:18.283385 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:18.283320 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:18.414945 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:18.414915 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:20.281909 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:20.281865 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:20.409557 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:20.409496 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:20.785590 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:20.785558 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:20.918768 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:20.918736 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:21.051120 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:21.051047 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:21.373698 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:21.373626 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:24.200347 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:24.200320 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jrfkr" Oct 31 14:46:24.527214 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:24.527184 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:24.661275 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:24.661207 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:25.250879 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:25.250847 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:25.389905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:25.389876 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:26.122361 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:26.122317 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:26.771460 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:26.771430 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:26.907855 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:26.907808 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:27.059388 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.059307 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:27.264006 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.263966 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:27.487666 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.487625 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:27.652017 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.651970 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:27.803239 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.803145 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:27.820325 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.820292 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pjdc4"] Oct 31 14:46:27.847302 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.847275 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.901343 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.901321 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 14:46:27.931098 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.931049 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:27.952348 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.952330 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\"" Oct 31 14:46:27.952348 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.952344 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 14:46:27.952558 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.952442 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 14:46:27.952558 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.952527 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 14:46:27.960259 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960237 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-tls\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960269 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-textfile\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960294 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960365 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrkl\" (UniqueName: \"kubernetes.io/projected/c06d05c5-fee2-433e-ab10-c368ebb51711-kube-api-access-bmrkl\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960446 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960606 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960514 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-root\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960606 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960577 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-wtmp\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960694 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960635 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06d05c5-fee2-433e-ab10-c368ebb51711-metrics-client-ca\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:27.960694 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:27.960660 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-sys\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.061788 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061724 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06d05c5-fee2-433e-ab10-c368ebb51711-metrics-client-ca\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.061788 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061756 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-sys\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.061788 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061777 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-tls\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061803 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-textfile\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061839 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061848 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-sys\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061856 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrkl\" (UniqueName: \"kubernetes.io/projected/c06d05c5-fee2-433e-ab10-c368ebb51711-kube-api-access-bmrkl\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061906 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061946 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-root\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.061986 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-wtmp\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062362 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.062036 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-root\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062362 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.062174 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-wtmp\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062362 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.062288 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-textfile\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062546 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.062503 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06d05c5-fee2-433e-ab10-c368ebb51711-metrics-client-ca\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.062603 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.062557 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.064294 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.064269 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-tls\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.064386 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.064308 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c06d05c5-fee2-433e-ab10-c368ebb51711-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.120549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.120523 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrkl\" (UniqueName: \"kubernetes.io/projected/c06d05c5-fee2-433e-ab10-c368ebb51711-kube-api-access-bmrkl\") pod \"node-exporter-pjdc4\" (UID: \"c06d05c5-fee2-433e-ab10-c368ebb51711\") " pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.130564 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.130530 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:28.192657 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.192630 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pjdc4" Oct 31 14:46:28.232962 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:46:28.232937 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06d05c5_fee2_433e_ab10_c368ebb51711.slice/crio-b061c48799ce8e047bfbfd423f5cdec2d6399bbf711e3f14b8b4ec8817549e6a WatchSource:0}: Error finding container b061c48799ce8e047bfbfd423f5cdec2d6399bbf711e3f14b8b4ec8817549e6a: Status 404 returned error can't find the container with id b061c48799ce8e047bfbfd423f5cdec2d6399bbf711e3f14b8b4ec8817549e6a Oct 31 14:46:28.529413 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.529378 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:28.712047 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:28.712011 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjdc4" event={"ID":"c06d05c5-fee2-433e-ab10-c368ebb51711","Type":"ContainerStarted","Data":"b061c48799ce8e047bfbfd423f5cdec2d6399bbf711e3f14b8b4ec8817549e6a"} Oct 31 14:46:29.126234 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:29.126187 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:29.327230 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:29.327199 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:29.837138 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:29.837110 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:29.963958 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:29.963928 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:30.104487 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:30.104393 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:30.262491 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:30.262446 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:30.409586 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:30.409501 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:30.725975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:30.725945 4865 generic.go:356] "Generic (PLEG): container finished" podID="c06d05c5-fee2-433e-ab10-c368ebb51711" containerID="0cc2959a115dbc07162a0f34c112817db1666501383c2e7bb304a21bd24bafc4" exitCode=0 Oct 31 14:46:30.726150 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:30.726002 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjdc4" event={"ID":"c06d05c5-fee2-433e-ab10-c368ebb51711","Type":"ContainerDied","Data":"0cc2959a115dbc07162a0f34c112817db1666501383c2e7bb304a21bd24bafc4"} Oct 31 14:46:31.731507 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:31.731453 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjdc4" event={"ID":"c06d05c5-fee2-433e-ab10-c368ebb51711","Type":"ContainerStarted","Data":"aacbcee9ca076e81be2f9a99dc465f53b549d18e3b3ae38a914c4ba92f5b48d2"} Oct 31 14:46:31.731901 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:31.731525 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjdc4" event={"ID":"c06d05c5-fee2-433e-ab10-c368ebb51711","Type":"ContainerStarted","Data":"0e2d86414c069fbdeb60148c3a94339524673f312175faf2076ec5ff82dc2a19"} Oct 31 14:46:31.879955 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:31.879922 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:31.900444 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:31.900392 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pjdc4" podStartSLOduration=3.312079132 podStartE2EDuration="4.900377707s" totalImagesPullingTime="1.511956155s" totalInitContainerRuntime="76.34242ms" isStatefulPod=true podCreationTimestamp="2026-10-31 14:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:31.899784456 +0000 UTC m=+37.519335867" watchObservedRunningTime="2026-10-31 14:46:31.900377707 +0000 UTC m=+37.519929117" Oct 31 14:46:32.019723 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:32.019633 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:32.147985 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:32.147949 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:32.278534 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:32.278451 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:32.415120 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:32.415089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:32.618344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:32.618261 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:33.176989 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:33.176952 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678477bddd-n4c79"] Oct 31 14:46:33.181538 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:33.181516 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:33.285095 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:33.285064 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:33.475084 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:33.475048 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:33.623634 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:33.623598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:36.764539 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:36.764435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:38.174564 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:38.174532 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:38.303164 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:38.303138 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:39.531100 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:39.531069 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:39.668291 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:39.668252 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:42.329266 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:42.329235 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:42.469375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:42.469340 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:44.465795 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:44.465767 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:44.665649 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:44.665621 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:46.385585 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:46.385559 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:46.516067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:46.516030 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:47.489081 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:47.489048 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:46:48.447712 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:48.447648 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:48.569083 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:48.569034 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:50.648491 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:50.648426 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:50.793357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:50.793326 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:54.709523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:54.709458 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:54.847496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:54.847423 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:46:58.197487 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.197430 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-678477bddd-n4c79" podUID="8340f44a-41ba-4624-9bc4-a7c8873ab788" containerName="registry" containerID="cri-o://89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07" gracePeriod=30 Oct 31 14:46:58.448568 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.448518 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:58.514656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.514628 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.514783 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.514678 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.514783 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.514714 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.514783 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.514744 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.514783 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.514767 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.514981 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.514957 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.515042 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.515009 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.515098 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.515045 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\") pod \"8340f44a-41ba-4624-9bc4-a7c8873ab788\" (UID: \"8340f44a-41ba-4624-9bc4-a7c8873ab788\") " Oct 31 14:46:58.515604 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.515165 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:46:58.515604 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.515567 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:46:58.515935 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.515638 4865 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-certificates\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.517431 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.517408 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:46:58.517589 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.517521 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "kube-api-access-2w2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:46:58.517700 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.517679 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:46:58.517762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.517707 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:46:58.517852 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.517834 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:46:58.521971 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.521950 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted" pod "8340f44a-41ba-4624-9bc4-a7c8873ab788" (UID: "8340f44a-41ba-4624-9bc4-a7c8873ab788"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:46:58.616449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.616426 4865 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8340f44a-41ba-4624-9bc4-a7c8873ab788-ca-trust-extracted\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.616449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.616446 4865 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-bound-sa-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.616449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.616456 4865 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-registry-tls\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.616665 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.616482 4865 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-image-registry-private-configuration\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.616665 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.616492 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w2f6\" (UniqueName: \"kubernetes.io/projected/8340f44a-41ba-4624-9bc4-a7c8873ab788-kube-api-access-2w2f6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.616665 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.616501 4865 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8340f44a-41ba-4624-9bc4-a7c8873ab788-trusted-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.616665 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.616510 4865 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8340f44a-41ba-4624-9bc4-a7c8873ab788-installation-pull-secrets\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:46:58.824263 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.824231 4865 generic.go:356] "Generic (PLEG): container finished" podID="8340f44a-41ba-4624-9bc4-a7c8873ab788" containerID="89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07" exitCode=0 Oct 31 14:46:58.824438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.824280 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678477bddd-n4c79" Oct 31 14:46:58.824438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.824321 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678477bddd-n4c79" event={"ID":"8340f44a-41ba-4624-9bc4-a7c8873ab788","Type":"ContainerDied","Data":"89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07"} Oct 31 14:46:58.824438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.824375 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678477bddd-n4c79" event={"ID":"8340f44a-41ba-4624-9bc4-a7c8873ab788","Type":"ContainerDied","Data":"d333d1c7b6d1fa81e9b37be8b4fd00483bdc8ffc89e0fa348bc3829fc1c394c4"} Oct 31 14:46:58.824438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.824401 4865 scope.go:122] "RemoveContainer" containerID="89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07" Oct 31 14:46:58.833313 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.833294 4865 scope.go:122] "RemoveContainer" containerID="89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07" Oct 31 14:46:58.833610 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:46:58.833592 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07\": container with ID starting with 89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07 not found: ID does not exist" containerID="89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07" Oct 31 14:46:58.833686 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.833623 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07"} err="failed to get container status \"89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07\": rpc error: code = NotFound desc = could not find container \"89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07\": container with ID starting with 89a8a046909f87f1989e71830ee5343112b34a39cf63f33cc6c1f29536658c07 not found: ID does not exist" Oct 31 14:46:58.990610 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.990577 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678477bddd-n4c79"] Oct 31 14:46:58.994949 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:46:58.994925 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-678477bddd-n4c79"] Oct 31 14:47:00.493895 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:00.493868 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8340f44a-41ba-4624-9bc4-a7c8873ab788" path="/var/lib/kubelet/pods/8340f44a-41ba-4624-9bc4-a7c8873ab788/volumes" Oct 31 14:47:01.908284 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:01.908248 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:02.033778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:02.033744 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:05.850027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:05.849994 4865 generic.go:356] "Generic (PLEG): container finished" podID="f3b0ed9d-d90a-497a-8863-7605a3c89c75" containerID="8bc1c92793439050e5bd14fd0dfdbab7305df888bc3d43fe9545b3590b4ff213" exitCode=0 Oct 31 14:47:05.850525 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:05.850041 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" event={"ID":"f3b0ed9d-d90a-497a-8863-7605a3c89c75","Type":"ContainerDied","Data":"8bc1c92793439050e5bd14fd0dfdbab7305df888bc3d43fe9545b3590b4ff213"} Oct 31 14:47:05.850525 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:05.850341 4865 scope.go:122] "RemoveContainer" containerID="8bc1c92793439050e5bd14fd0dfdbab7305df888bc3d43fe9545b3590b4ff213" Oct 31 14:47:06.411087 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:06.411058 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:06.539894 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:06.539862 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:06.856077 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:06.856039 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-sxs2m" event={"ID":"f3b0ed9d-d90a-497a-8863-7605a3c89c75","Type":"ContainerStarted","Data":"158577fa7e318eff3a9b92113c23b057e0498485d5f2291a483610c1072493d3"} Oct 31 14:47:09.870190 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:09.870160 4865 generic.go:356] "Generic (PLEG): container finished" podID="f9259fbe-036b-4629-a947-2dcd24185c78" containerID="6d9305c38f7284dbc5dc3201d85197a3188a71ee91eaf0564e4f4a27c7ce5d9b" exitCode=0 Oct 31 14:47:09.870652 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:09.870199 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-n4mj8" event={"ID":"f9259fbe-036b-4629-a947-2dcd24185c78","Type":"ContainerDied","Data":"6d9305c38f7284dbc5dc3201d85197a3188a71ee91eaf0564e4f4a27c7ce5d9b"} Oct 31 14:47:09.870652 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:09.870537 4865 scope.go:122] "RemoveContainer" containerID="6d9305c38f7284dbc5dc3201d85197a3188a71ee91eaf0564e4f4a27c7ce5d9b" Oct 31 14:47:10.881387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:10.881344 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-n4mj8" event={"ID":"f9259fbe-036b-4629-a947-2dcd24185c78","Type":"ContainerStarted","Data":"89fdf1404b9f28e00ce2050e142eb461813adc466d22328f68da9f354bb3e252"} Oct 31 14:47:10.882865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:10.882845 4865 generic.go:356] "Generic (PLEG): container finished" podID="58930f87-6f50-486e-9ab6-b58f7aee05c2" containerID="c9125bd1ae51819fe329b8f5b34c307b3e44db08998e17bbad4528820e263843" exitCode=0 Oct 31 14:47:10.882947 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:10.882897 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" event={"ID":"58930f87-6f50-486e-9ab6-b58f7aee05c2","Type":"ContainerDied","Data":"c9125bd1ae51819fe329b8f5b34c307b3e44db08998e17bbad4528820e263843"} Oct 31 14:47:10.883163 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:10.883150 4865 scope.go:122] "RemoveContainer" containerID="c9125bd1ae51819fe329b8f5b34c307b3e44db08998e17bbad4528820e263843" Oct 31 14:47:11.888419 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:11.888385 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hmvx2" event={"ID":"58930f87-6f50-486e-9ab6-b58f7aee05c2","Type":"ContainerStarted","Data":"58c66619cde97d82143423fde3a0c9e09f59a275296a59061905e2f08f1d81f5"} Oct 31 14:47:14.508238 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:14.508205 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:14.629136 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:14.629107 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:17.488896 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:47:17.488856 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:47:18.188446 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:18.188408 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:18.586554 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:18.586523 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:18.717871 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:18.717817 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:27.441106 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:27.441074 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:27.573148 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:27.573120 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:39.534909 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:39.534867 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:39.665855 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:39.665826 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:44.845635 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:47:44.845605 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:47:47.489364 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:47:47.489329 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:48:11.250715 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:11.250633 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:11.470529 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:11.470493 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:12.665362 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.665330 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq"] Oct 31 14:48:12.665769 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.665638 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8340f44a-41ba-4624-9bc4-a7c8873ab788" containerName="registry" Oct 31 14:48:12.665769 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.665649 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8340f44a-41ba-4624-9bc4-a7c8873ab788" containerName="registry" Oct 31 14:48:12.665769 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.665712 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8340f44a-41ba-4624-9bc4-a7c8873ab788" containerName="registry" Oct 31 14:48:12.668767 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.668751 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.722304 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.722276 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 14:48:12.722463 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.722351 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 14:48:12.757903 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.757869 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmfd\" (UniqueName: \"kubernetes.io/projected/9db6fa18-a893-41f2-8073-01a123a7846e-kube-api-access-fqmfd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.758049 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.757989 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.758049 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.758033 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.776355 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.774137 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-n42mj\"" Oct 31 14:48:12.776355 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.776231 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq"] Oct 31 14:48:12.859012 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.858977 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.859180 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.859034 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.859180 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.859080 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmfd\" (UniqueName: \"kubernetes.io/projected/9db6fa18-a893-41f2-8073-01a123a7846e-kube-api-access-fqmfd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.859484 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.859440 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.859598 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.859575 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.904845 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.904823 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922"] Oct 31 14:48:12.907990 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.907976 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.917523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.917461 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmfd\" (UniqueName: \"kubernetes.io/projected/9db6fa18-a893-41f2-8073-01a123a7846e-kube-api-access-fqmfd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:12.940026 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.939972 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:12.960011 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.959988 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6d9n\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-kube-api-access-w6d9n\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.960093 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.960020 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.960093 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.960053 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.960180 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.960158 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.960233 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.960182 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/217ef3bf-f7b3-4eca-bb90-a45edb46be17-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.960233 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.960199 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.960334 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.960264 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:12.971012 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.970992 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 14:48:12.971867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.971846 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 14:48:12.971966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.971884 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 14:48:12.972030 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.971981 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 14:48:12.972131 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.972113 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jwmrt\"" Oct 31 14:48:12.972631 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.972608 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922"] Oct 31 14:48:12.984232 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:12.984210 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:13.061508 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.061461 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.061669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.061521 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.061669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.061547 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/217ef3bf-f7b3-4eca-bb90-a45edb46be17-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.061669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.061566 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.061669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.061592 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.061669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.061629 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6d9n\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-kube-api-access-w6d9n\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.061669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.061659 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.062732 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.062704 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.064785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.064743 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.064785 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.064771 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/217ef3bf-f7b3-4eca-bb90-a45edb46be17-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.064956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.064890 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.065018 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.064998 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.087003 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.086980 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:13.134812 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.134781 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6d9n\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-kube-api-access-w6d9n\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.135160 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.135142 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-68922\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.219102 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.219079 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:13.354153 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.354132 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq"] Oct 31 14:48:13.356531 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:48:13.356504 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db6fa18_a893_41f2_8073_01a123a7846e.slice/crio-c469a951e481beb058f70fc31f67a3f3ef94c90d2fe4c4e5563c917b0bab19b6 WatchSource:0}: Error finding container c469a951e481beb058f70fc31f67a3f3ef94c90d2fe4c4e5563c917b0bab19b6: Status 404 returned error can't find the container with id c469a951e481beb058f70fc31f67a3f3ef94c90d2fe4c4e5563c917b0bab19b6 Oct 31 14:48:13.589401 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:13.589381 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922"] Oct 31 14:48:13.591725 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:48:13.591699 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217ef3bf_f7b3_4eca_bb90_a45edb46be17.slice/crio-c4ca99d0d7563a24c7e20281869d06b6fad05c0d78651c37e518608d9e2b2cca WatchSource:0}: Error finding container c4ca99d0d7563a24c7e20281869d06b6fad05c0d78651c37e518608d9e2b2cca: Status 404 returned error can't find the container with id c4ca99d0d7563a24c7e20281869d06b6fad05c0d78651c37e518608d9e2b2cca Oct 31 14:48:14.112184 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:14.112153 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" event={"ID":"217ef3bf-f7b3-4eca-bb90-a45edb46be17","Type":"ContainerStarted","Data":"c4ca99d0d7563a24c7e20281869d06b6fad05c0d78651c37e518608d9e2b2cca"} Oct 31 14:48:14.113147 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:14.113127 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" event={"ID":"9db6fa18-a893-41f2-8073-01a123a7846e","Type":"ContainerStarted","Data":"c469a951e481beb058f70fc31f67a3f3ef94c90d2fe4c4e5563c917b0bab19b6"} Oct 31 14:48:14.224190 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:14.224158 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:17.493972 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:48:17.493940 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:48:17.526713 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:17.526677 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:48:17.526834 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:17.526761 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:48:18.133917 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:18.133878 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" event={"ID":"217ef3bf-f7b3-4eca-bb90-a45edb46be17","Type":"ContainerStarted","Data":"79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1"} Oct 31 14:48:18.134120 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:18.134068 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:18.136173 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:18.136131 4865 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-68922 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:48:18.136396 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:18.136359 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:48:18.357292 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:18.357237 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" podStartSLOduration=2.425176058 podStartE2EDuration="6.357222452s" totalImagesPullingTime="3.932046394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:48:18.306033201 +0000 UTC m=+143.925584615" watchObservedRunningTime="2026-10-31 14:48:18.357222452 +0000 UTC m=+143.976773863" Oct 31 14:48:18.933377 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:18.933344 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:19.106398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:19.106323 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:19.139242 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:19.139198 4865 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-68922 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:48:19.139396 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:19.139261 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:48:19.973740 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:19.973719 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:20.143966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:20.143937 4865 generic.go:356] "Generic (PLEG): container finished" podID="9db6fa18-a893-41f2-8073-01a123a7846e" containerID="0b51944dd1bba13c4fae88796d9b35b207de5b74639a7bfd05f3465f7ff56116" exitCode=0 Oct 31 14:48:20.144152 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:20.144034 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" event={"ID":"9db6fa18-a893-41f2-8073-01a123a7846e","Type":"ContainerDied","Data":"0b51944dd1bba13c4fae88796d9b35b207de5b74639a7bfd05f3465f7ff56116"} Oct 31 14:48:20.202829 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:20.202759 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:20.349589 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:20.349553 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:22.139375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:22.139344 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:48:23.867875 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:23.867839 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:24.161179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:24.161103 4865 generic.go:356] "Generic (PLEG): container finished" podID="9db6fa18-a893-41f2-8073-01a123a7846e" containerID="589ee562830c53c9441a62b96053b6b568d5ccf9cc77bd33e3bff3bf33139a91" exitCode=0 Oct 31 14:48:24.161179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:24.161163 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" event={"ID":"9db6fa18-a893-41f2-8073-01a123a7846e","Type":"ContainerDied","Data":"589ee562830c53c9441a62b96053b6b568d5ccf9cc77bd33e3bff3bf33139a91"} Oct 31 14:48:31.192518 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:31.192420 4865 generic.go:356] "Generic (PLEG): container finished" podID="9db6fa18-a893-41f2-8073-01a123a7846e" containerID="748324bcca6129b6a9216720eb7f638d75f52646b5b702239b28ae3957266832" exitCode=0 Oct 31 14:48:31.192860 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:31.192510 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" event={"ID":"9db6fa18-a893-41f2-8073-01a123a7846e","Type":"ContainerDied","Data":"748324bcca6129b6a9216720eb7f638d75f52646b5b702239b28ae3957266832"} Oct 31 14:48:32.333583 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.333560 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:32.428693 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.428660 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db6fa18-a893-41f2-8073-01a123a7846e-kube-api-access-fqmfd\" (UniqueName: \"kubernetes.io/projected/9db6fa18-a893-41f2-8073-01a123a7846e-kube-api-access-fqmfd\") pod \"9db6fa18-a893-41f2-8073-01a123a7846e\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " Oct 31 14:48:32.428848 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.428702 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-bundle\") pod \"9db6fa18-a893-41f2-8073-01a123a7846e\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " Oct 31 14:48:32.428848 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.428783 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-util\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-util\") pod \"9db6fa18-a893-41f2-8073-01a123a7846e\" (UID: \"9db6fa18-a893-41f2-8073-01a123a7846e\") " Oct 31 14:48:32.429170 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.429143 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-bundle" pod "9db6fa18-a893-41f2-8073-01a123a7846e" (UID: "9db6fa18-a893-41f2-8073-01a123a7846e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:48:32.430998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.430979 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db6fa18-a893-41f2-8073-01a123a7846e-kube-api-access-fqmfd" pod "9db6fa18-a893-41f2-8073-01a123a7846e" (UID: "9db6fa18-a893-41f2-8073-01a123a7846e"). InnerVolumeSpecName "kube-api-access-fqmfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:48:32.433280 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.433260 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-util" pod "9db6fa18-a893-41f2-8073-01a123a7846e" (UID: "9db6fa18-a893-41f2-8073-01a123a7846e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:48:32.530195 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.530169 4865 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-util\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:48:32.530195 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.530192 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmfd\" (UniqueName: \"kubernetes.io/projected/9db6fa18-a893-41f2-8073-01a123a7846e-kube-api-access-fqmfd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:48:32.530357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:32.530202 4865 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9db6fa18-a893-41f2-8073-01a123a7846e-bundle\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:48:33.202033 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:33.202001 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" event={"ID":"9db6fa18-a893-41f2-8073-01a123a7846e","Type":"ContainerDied","Data":"c469a951e481beb058f70fc31f67a3f3ef94c90d2fe4c4e5563c917b0bab19b6"} Oct 31 14:48:33.202241 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:33.202054 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zfvpq" Oct 31 14:48:33.202241 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:33.202070 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c469a951e481beb058f70fc31f67a3f3ef94c90d2fe4c4e5563c917b0bab19b6" Oct 31 14:48:45.190415 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:48:45.190385 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:48:47.488159 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:48:47.488119 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:49:00.629714 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:00.629682 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:09.051490 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.051438 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:09.261756 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.261719 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:09.683942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.683910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.994701 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw"] Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.995279 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9db6fa18-a893-41f2-8073-01a123a7846e" containerName="extract" Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.995296 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db6fa18-a893-41f2-8073-01a123a7846e" containerName="extract" Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.995343 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9db6fa18-a893-41f2-8073-01a123a7846e" containerName="util" Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.995353 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db6fa18-a893-41f2-8073-01a123a7846e" containerName="util" Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.995379 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9db6fa18-a893-41f2-8073-01a123a7846e" containerName="pull" Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.995386 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db6fa18-a893-41f2-8073-01a123a7846e" containerName="pull" Oct 31 14:49:09.996398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.995543 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db6fa18-a893-41f2-8073-01a123a7846e" containerName="extract" Oct 31 14:49:09.998254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:09.998232 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.060452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.060423 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:10.106385 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.106354 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw"] Oct 31 14:49:10.109593 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.109564 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.109714 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.109636 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.109763 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.109706 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.109763 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.109750 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.109861 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.109771 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.109914 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.109896 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvjp\" (UniqueName: \"kubernetes.io/projected/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-kube-api-access-rnvjp\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.156223 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.156195 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 14:49:10.156223 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.156219 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 14:49:10.156425 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.156226 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 14:49:10.156425 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.156380 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 14:49:10.156560 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.156426 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 14:49:10.156560 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.156508 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\"" Oct 31 14:49:10.207619 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.207590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:10.210740 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.210723 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.210801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.210748 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.210801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.210769 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvjp\" (UniqueName: \"kubernetes.io/projected/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-kube-api-access-rnvjp\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.210801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.210797 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.210944 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.210825 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.210944 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.210844 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.211252 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.211226 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.213229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.213203 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.213423 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.213407 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.213510 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.213489 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.213555 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.213532 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.270461 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.270398 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvjp\" (UniqueName: \"kubernetes.io/projected/7fea1876-d46d-42bf-96f4-f1fef8a1b6fa-kube-api-access-rnvjp\") pod \"opendatahub-operator-controller-manager-c95c77854-jzhxw\" (UID: \"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.311413 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.311393 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:10.350659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.350613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:10.801332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:10.801307 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw"] Oct 31 14:49:10.802942 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:49:10.802916 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fea1876_d46d_42bf_96f4_f1fef8a1b6fa.slice/crio-cc2825558f5bbe459fe2c363e0e8d2b3031cb9057c0fffb4748020e160c8d6b4 WatchSource:0}: Error finding container cc2825558f5bbe459fe2c363e0e8d2b3031cb9057c0fffb4748020e160c8d6b4: Status 404 returned error can't find the container with id cc2825558f5bbe459fe2c363e0e8d2b3031cb9057c0fffb4748020e160c8d6b4 Oct 31 14:49:11.218808 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:11.218777 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:11.343442 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:11.343411 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" event={"ID":"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa","Type":"ContainerStarted","Data":"cc2825558f5bbe459fe2c363e0e8d2b3031cb9057c0fffb4748020e160c8d6b4"} Oct 31 14:49:11.383377 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:11.383342 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:12.666660 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:12.666632 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:12.815131 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:12.815089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:13.050666 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.050630 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7"] Oct 31 14:49:13.055433 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.055407 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.112384 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.112346 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7"] Oct 31 14:49:13.135288 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.135257 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.135499 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.135304 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.135499 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.135392 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnf4\" (UniqueName: \"kubernetes.io/projected/a6fa1f88-02b8-4576-908f-4505a45c785e-kube-api-access-stnf4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.162270 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.162237 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 14:49:13.162445 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.162276 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ddl84\"" Oct 31 14:49:13.162445 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.162299 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 14:49:13.236732 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.236702 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.236898 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.236741 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.236898 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.236763 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnf4\" (UniqueName: \"kubernetes.io/projected/a6fa1f88-02b8-4576-908f-4505a45c785e-kube-api-access-stnf4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.237154 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.237135 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.237247 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.237223 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.295449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.295417 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnf4\" (UniqueName: \"kubernetes.io/projected/a6fa1f88-02b8-4576-908f-4505a45c785e-kube-api-access-stnf4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.370888 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.370804 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:13.453526 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.453488 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:13.589745 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.589709 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:13.757246 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:13.757223 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7"] Oct 31 14:49:13.760041 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:49:13.760010 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fa1f88_02b8_4576_908f_4505a45c785e.slice/crio-c27abf92996adf39a6ccb27c2f2bf1b0b4cc6a56aa85488861300ca3a156f248 WatchSource:0}: Error finding container c27abf92996adf39a6ccb27c2f2bf1b0b4cc6a56aa85488861300ca3a156f248: Status 404 returned error can't find the container with id c27abf92996adf39a6ccb27c2f2bf1b0b4cc6a56aa85488861300ca3a156f248 Oct 31 14:49:14.357735 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:14.357703 4865 generic.go:356] "Generic (PLEG): container finished" podID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerID="9675328ccce117efc5c9f1450e8752ac8d30ff1a9c74e7f63d4877ab9d0e6a44" exitCode=0 Oct 31 14:49:14.357888 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:14.357787 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" event={"ID":"a6fa1f88-02b8-4576-908f-4505a45c785e","Type":"ContainerDied","Data":"9675328ccce117efc5c9f1450e8752ac8d30ff1a9c74e7f63d4877ab9d0e6a44"} Oct 31 14:49:14.357888 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:14.357840 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" event={"ID":"a6fa1f88-02b8-4576-908f-4505a45c785e","Type":"ContainerStarted","Data":"c27abf92996adf39a6ccb27c2f2bf1b0b4cc6a56aa85488861300ca3a156f248"} Oct 31 14:49:14.359577 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:14.359557 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" event={"ID":"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa","Type":"ContainerStarted","Data":"8d89ffc80e9d330d5d1d910ecb87c54f77b919cdf6d517c37166e049e0e7a3a2"} Oct 31 14:49:15.365986 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:15.365952 4865 generic.go:356] "Generic (PLEG): container finished" podID="7fea1876-d46d-42bf-96f4-f1fef8a1b6fa" containerID="8d89ffc80e9d330d5d1d910ecb87c54f77b919cdf6d517c37166e049e0e7a3a2" exitCode=0 Oct 31 14:49:15.366407 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:15.366037 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" event={"ID":"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa","Type":"ContainerDied","Data":"8d89ffc80e9d330d5d1d910ecb87c54f77b919cdf6d517c37166e049e0e7a3a2"} Oct 31 14:49:16.372552 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:16.372521 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" event={"ID":"7fea1876-d46d-42bf-96f4-f1fef8a1b6fa","Type":"ContainerStarted","Data":"fabee8ae4e98eb4d9facdee3be49558ec0ec1963669fced5a150735817cf37b1"} Oct 31 14:49:16.372922 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:16.372632 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:16.374313 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:16.374293 4865 generic.go:356] "Generic (PLEG): container finished" podID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerID="acd437b3cd3ededb17731868c85c444f66ccc41f8efbc1a1b656ba1307f6ab8c" exitCode=0 Oct 31 14:49:16.374390 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:16.374345 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" event={"ID":"a6fa1f88-02b8-4576-908f-4505a45c785e","Type":"ContainerDied","Data":"acd437b3cd3ededb17731868c85c444f66ccc41f8efbc1a1b656ba1307f6ab8c"} Oct 31 14:49:16.576301 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:16.576207 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" podStartSLOduration=2.9028876 podStartE2EDuration="7.57619263s" totalImagesPullingTime="4.261271883s" totalInitContainerRuntime="412.033147ms" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:16.543652028 +0000 UTC m=+202.163203452" watchObservedRunningTime="2026-10-31 14:49:16.57619263 +0000 UTC m=+202.195744096" Oct 31 14:49:17.381012 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:17.380928 4865 generic.go:356] "Generic (PLEG): container finished" podID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerID="1c972a99a6e5c8d725f2cea582c837ddba0d0b8fb7914caf878691396ec79a94" exitCode=0 Oct 31 14:49:17.381366 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:17.381012 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" event={"ID":"a6fa1f88-02b8-4576-908f-4505a45c785e","Type":"ContainerDied","Data":"1c972a99a6e5c8d725f2cea582c837ddba0d0b8fb7914caf878691396ec79a94"} Oct 31 14:49:17.487670 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:49:17.487637 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:49:17.690698 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:17.690622 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:18.511925 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.511897 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:18.581360 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.581329 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-bundle\") pod \"a6fa1f88-02b8-4576-908f-4505a45c785e\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " Oct 31 14:49:18.581543 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.581387 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-util\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-util\") pod \"a6fa1f88-02b8-4576-908f-4505a45c785e\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " Oct 31 14:49:18.581543 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.581432 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fa1f88-02b8-4576-908f-4505a45c785e-kube-api-access-stnf4\" (UniqueName: \"kubernetes.io/projected/a6fa1f88-02b8-4576-908f-4505a45c785e-kube-api-access-stnf4\") pod \"a6fa1f88-02b8-4576-908f-4505a45c785e\" (UID: \"a6fa1f88-02b8-4576-908f-4505a45c785e\") " Oct 31 14:49:18.581959 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.581936 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-bundle" pod "a6fa1f88-02b8-4576-908f-4505a45c785e" (UID: "a6fa1f88-02b8-4576-908f-4505a45c785e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:49:18.583646 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.583621 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fa1f88-02b8-4576-908f-4505a45c785e-kube-api-access-stnf4" pod "a6fa1f88-02b8-4576-908f-4505a45c785e" (UID: "a6fa1f88-02b8-4576-908f-4505a45c785e"). InnerVolumeSpecName "kube-api-access-stnf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:49:18.588959 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.588930 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-util" pod "a6fa1f88-02b8-4576-908f-4505a45c785e" (UID: "a6fa1f88-02b8-4576-908f-4505a45c785e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:49:18.682495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.682398 4865 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-util\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:49:18.682495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.682427 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnf4\" (UniqueName: \"kubernetes.io/projected/a6fa1f88-02b8-4576-908f-4505a45c785e-kube-api-access-stnf4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:49:18.682495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:18.682437 4865 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6fa1f88-02b8-4576-908f-4505a45c785e-bundle\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:49:19.391756 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:19.391721 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" event={"ID":"a6fa1f88-02b8-4576-908f-4505a45c785e","Type":"ContainerDied","Data":"c27abf92996adf39a6ccb27c2f2bf1b0b4cc6a56aa85488861300ca3a156f248"} Oct 31 14:49:19.391966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:19.391765 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27abf92996adf39a6ccb27c2f2bf1b0b4cc6a56aa85488861300ca3a156f248" Oct 31 14:49:19.391966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:19.391764 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atwmp7" Oct 31 14:49:21.776025 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:21.775990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:23.432062 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:23.432028 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:23.562331 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:23.562302 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:26.452905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:26.452874 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:26.843271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:26.843236 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:27.017124 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:27.017088 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:27.383371 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:27.383345 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-jzhxw" Oct 31 14:49:27.425112 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:27.425077 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:29.135866 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:29.135835 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:29.645730 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:29.645697 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:32.898051 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:32.898015 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:33.235088 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:33.235057 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:33.365705 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:33.365667 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:33.641001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:33.640927 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:34.531001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:34.530965 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:34.777235 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:34.777203 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:36.020338 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:36.020268 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:36.148417 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:36.148381 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:36.933977 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:36.933948 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:38.743459 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:38.743430 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:38.877845 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:38.877807 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:40.047907 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:40.047880 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:41.702149 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:41.702106 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:43.255498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:43.255451 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:43.552111 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:43.552023 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:44.079950 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:44.079917 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:44.261405 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:44.261372 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:44.697519 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:44.697486 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:45.614684 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:45.614652 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:46.344119 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.344088 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:46.379459 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379428 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9d7844885-gnrpl"] Oct 31 14:49:46.379768 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379755 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerName="pull" Oct 31 14:49:46.379827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379771 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerName="pull" Oct 31 14:49:46.379827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379786 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerName="extract" Oct 31 14:49:46.379827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379790 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerName="extract" Oct 31 14:49:46.379827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379809 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerName="util" Oct 31 14:49:46.379827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379814 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerName="util" Oct 31 14:49:46.380011 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.379868 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fa1f88-02b8-4576-908f-4505a45c785e" containerName="extract" Oct 31 14:49:46.385144 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.385126 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.493752 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.493716 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9d7844885-gnrpl"] Oct 31 14:49:46.514673 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.514642 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgsm\" (UniqueName: \"kubernetes.io/projected/3c7543a2-6d1f-4a11-821c-2cd20444239e-kube-api-access-mdgsm\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.514673 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.514676 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c7543a2-6d1f-4a11-821c-2cd20444239e-tmp\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.514905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.514706 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7543a2-6d1f-4a11-821c-2cd20444239e-tls-certs\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.542344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.542311 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsph8\"" Oct 31 14:49:46.542344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.542316 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:46.542344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.542311 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 14:49:46.553213 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.553193 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 14:49:46.615637 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.615560 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7543a2-6d1f-4a11-821c-2cd20444239e-tls-certs\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.616101 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.615666 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgsm\" (UniqueName: \"kubernetes.io/projected/3c7543a2-6d1f-4a11-821c-2cd20444239e-kube-api-access-mdgsm\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.616101 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.615698 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c7543a2-6d1f-4a11-821c-2cd20444239e-tmp\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.618126 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.618102 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c7543a2-6d1f-4a11-821c-2cd20444239e-tmp\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.618357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.618338 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7543a2-6d1f-4a11-821c-2cd20444239e-tls-certs\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.677653 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.677622 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgsm\" (UniqueName: \"kubernetes.io/projected/3c7543a2-6d1f-4a11-821c-2cd20444239e-kube-api-access-mdgsm\") pod \"kube-auth-proxy-9d7844885-gnrpl\" (UID: \"3c7543a2-6d1f-4a11-821c-2cd20444239e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.687364 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.687340 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:46.697344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.697313 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" Oct 31 14:49:46.871313 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:46.871231 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:47.022619 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:47.022589 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:47.074332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:47.074310 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9d7844885-gnrpl"] Oct 31 14:49:47.076753 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:49:47.076728 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7543a2_6d1f_4a11_821c_2cd20444239e.slice/crio-4ced8b2b283f346a4304e95f9ce6ff3b8eb39d23be15f48360601c199672cdd4 WatchSource:0}: Error finding container 4ced8b2b283f346a4304e95f9ce6ff3b8eb39d23be15f48360601c199672cdd4: Status 404 returned error can't find the container with id 4ced8b2b283f346a4304e95f9ce6ff3b8eb39d23be15f48360601c199672cdd4 Oct 31 14:49:47.223420 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:47.223396 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:47.488746 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:49:47.488671 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:49:47.502730 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:47.502701 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" event={"ID":"3c7543a2-6d1f-4a11-821c-2cd20444239e","Type":"ContainerStarted","Data":"4ced8b2b283f346a4304e95f9ce6ff3b8eb39d23be15f48360601c199672cdd4"} Oct 31 14:49:50.518190 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:50.518149 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" event={"ID":"3c7543a2-6d1f-4a11-821c-2cd20444239e","Type":"ContainerStarted","Data":"d7fb4295dc2f49b055617f5cb490e070379fbb84208b5e0eb08a7d4474ef085e"} Oct 31 14:49:50.636745 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:50.636696 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9d7844885-gnrpl" podStartSLOduration=1.8207410130000001 podStartE2EDuration="4.636680436s" totalImagesPullingTime="2.815939423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:50.6356787 +0000 UTC m=+236.255230111" watchObservedRunningTime="2026-10-31 14:49:50.636680436 +0000 UTC m=+236.256231847" Oct 31 14:49:52.221409 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:52.221372 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:52.351092 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:52.351039 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:52.513350 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:52.513246 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:52.670347 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:52.670303 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:54.316772 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:54.316733 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:58.528862 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:58.528831 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:58.652207 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:58.652170 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:58.775175 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:58.775144 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:58.907163 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:58.907053 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:49:59.038245 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:49:59.038174 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:00.464085 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:00.464053 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:00.594578 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:00.594546 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:02.006704 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:02.006672 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:02.377397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:02.377319 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:02.510407 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:02.510374 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:03.255275 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.255250 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:03.384611 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.384574 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:03.568088 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.568013 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lgtlw"] Oct 31 14:50:03.570667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.570649 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" Oct 31 14:50:03.626302 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.626279 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lgtlw"] Oct 31 14:50:03.651042 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.651017 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q52f\" (UniqueName: \"kubernetes.io/projected/e14af4c4-4ff7-4aa4-8586-f021c0abfc97-kube-api-access-8q52f\") pod \"authorino-57f7f8659f-lgtlw\" (UID: \"e14af4c4-4ff7-4aa4-8586-f021c0abfc97\") " pod="kuadrant-system/authorino-57f7f8659f-lgtlw" Oct 31 14:50:03.675387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.675360 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 14:50:03.675387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.675373 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 14:50:03.675569 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.675395 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lr8w6\"" Oct 31 14:50:03.733612 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.733584 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:03.751905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.751873 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q52f\" (UniqueName: \"kubernetes.io/projected/e14af4c4-4ff7-4aa4-8586-f021c0abfc97-kube-api-access-8q52f\") pod \"authorino-57f7f8659f-lgtlw\" (UID: \"e14af4c4-4ff7-4aa4-8586-f021c0abfc97\") " pod="kuadrant-system/authorino-57f7f8659f-lgtlw" Oct 31 14:50:03.810607 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.810582 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q52f\" (UniqueName: \"kubernetes.io/projected/e14af4c4-4ff7-4aa4-8586-f021c0abfc97-kube-api-access-8q52f\") pod \"authorino-57f7f8659f-lgtlw\" (UID: \"e14af4c4-4ff7-4aa4-8586-f021c0abfc97\") " pod="kuadrant-system/authorino-57f7f8659f-lgtlw" Oct 31 14:50:03.863860 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.863786 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:03.898897 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:03.898875 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" Oct 31 14:50:04.265484 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:04.265440 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lgtlw"] Oct 31 14:50:04.267442 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:50:04.267409 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode14af4c4_4ff7_4aa4_8586_f021c0abfc97.slice/crio-ac82464b0de5b90299f94826d8becec1c7ce62d9c717a7052fb57b72ea946239 WatchSource:0}: Error finding container ac82464b0de5b90299f94826d8becec1c7ce62d9c717a7052fb57b72ea946239: Status 404 returned error can't find the container with id ac82464b0de5b90299f94826d8becec1c7ce62d9c717a7052fb57b72ea946239 Oct 31 14:50:04.575489 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:04.575379 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" event={"ID":"e14af4c4-4ff7-4aa4-8586-f021c0abfc97","Type":"ContainerStarted","Data":"ac82464b0de5b90299f94826d8becec1c7ce62d9c717a7052fb57b72ea946239"} Oct 31 14:50:06.153262 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.153232 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:50:06.156814 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.156794 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.171834 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.171807 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjjm\" (UniqueName: \"kubernetes.io/projected/cd1efbf8-d417-453e-a9e4-0be4b5b6d393-kube-api-access-fgjjm\") pod \"limitador-limitador-7c9b87d574-2f7n9\" (UID: \"cd1efbf8-d417-453e-a9e4-0be4b5b6d393\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.171940 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.171866 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd1efbf8-d417-453e-a9e4-0be4b5b6d393-config-file\") pod \"limitador-limitador-7c9b87d574-2f7n9\" (UID: \"cd1efbf8-d417-453e-a9e4-0be4b5b6d393\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.212755 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.212727 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:50:06.261831 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.261807 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 14:50:06.261972 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.261850 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xtrmf\"" Oct 31 14:50:06.272888 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.272864 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjjm\" (UniqueName: \"kubernetes.io/projected/cd1efbf8-d417-453e-a9e4-0be4b5b6d393-kube-api-access-fgjjm\") pod \"limitador-limitador-7c9b87d574-2f7n9\" (UID: \"cd1efbf8-d417-453e-a9e4-0be4b5b6d393\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.273023 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.272964 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd1efbf8-d417-453e-a9e4-0be4b5b6d393-config-file\") pod \"limitador-limitador-7c9b87d574-2f7n9\" (UID: \"cd1efbf8-d417-453e-a9e4-0be4b5b6d393\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.273688 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.273666 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd1efbf8-d417-453e-a9e4-0be4b5b6d393-config-file\") pod \"limitador-limitador-7c9b87d574-2f7n9\" (UID: \"cd1efbf8-d417-453e-a9e4-0be4b5b6d393\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.334601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.334573 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjjm\" (UniqueName: \"kubernetes.io/projected/cd1efbf8-d417-453e-a9e4-0be4b5b6d393-kube-api-access-fgjjm\") pod \"limitador-limitador-7c9b87d574-2f7n9\" (UID: \"cd1efbf8-d417-453e-a9e4-0be4b5b6d393\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.402899 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.402869 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:50:06.469862 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.469550 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:06.852899 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:06.852875 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:50:06.854772 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:50:06.854740 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1efbf8_d417_453e_a9e4_0be4b5b6d393.slice/crio-65f70f6e0b7fa0518107820705b262ee1da9f659c731c7ddbeaa6f0d8700776a WatchSource:0}: Error finding container 65f70f6e0b7fa0518107820705b262ee1da9f659c731c7ddbeaa6f0d8700776a: Status 404 returned error can't find the container with id 65f70f6e0b7fa0518107820705b262ee1da9f659c731c7ddbeaa6f0d8700776a Oct 31 14:50:07.067728 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:07.067691 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:07.213356 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:07.213327 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:07.591449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:07.591367 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" event={"ID":"cd1efbf8-d417-453e-a9e4-0be4b5b6d393","Type":"ContainerStarted","Data":"65f70f6e0b7fa0518107820705b262ee1da9f659c731c7ddbeaa6f0d8700776a"} Oct 31 14:50:07.654287 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:07.654232 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:07.838899 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:07.838866 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:08.597324 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:08.597294 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" event={"ID":"e14af4c4-4ff7-4aa4-8586-f021c0abfc97","Type":"ContainerStarted","Data":"eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449"} Oct 31 14:50:08.598882 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:08.598862 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" event={"ID":"cd1efbf8-d417-453e-a9e4-0be4b5b6d393","Type":"ContainerStarted","Data":"725c865e884218a85212565e841acc01ad4b58d255584e4a613228138ffd2922"} Oct 31 14:50:08.598975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:08.598904 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:08.713347 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:08.713283 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" podStartSLOduration=1.690024821 podStartE2EDuration="5.713269746s" totalImagesPullingTime="4.023244925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:08.711101213 +0000 UTC m=+254.330652628" watchObservedRunningTime="2026-10-31 14:50:08.713269746 +0000 UTC m=+254.332821157" Oct 31 14:50:08.830341 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:08.830248 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" podStartSLOduration=1.1647366909999999 podStartE2EDuration="2.830234233s" totalImagesPullingTime="1.665497542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:08.829585867 +0000 UTC m=+254.449137279" watchObservedRunningTime="2026-10-31 14:50:08.830234233 +0000 UTC m=+254.449785644" Oct 31 14:50:10.075462 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:10.075424 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lgtlw"] Oct 31 14:50:10.607232 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:10.607198 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" podUID="e14af4c4-4ff7-4aa4-8586-f021c0abfc97" containerName="authorino" containerID="cri-o://eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449" gracePeriod=30 Oct 31 14:50:10.843829 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:10.843803 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" Oct 31 14:50:10.913237 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:10.913175 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14af4c4-4ff7-4aa4-8586-f021c0abfc97-kube-api-access-8q52f\" (UniqueName: \"kubernetes.io/projected/e14af4c4-4ff7-4aa4-8586-f021c0abfc97-kube-api-access-8q52f\") pod \"e14af4c4-4ff7-4aa4-8586-f021c0abfc97\" (UID: \"e14af4c4-4ff7-4aa4-8586-f021c0abfc97\") " Oct 31 14:50:10.915429 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:10.915406 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14af4c4-4ff7-4aa4-8586-f021c0abfc97-kube-api-access-8q52f" pod "e14af4c4-4ff7-4aa4-8586-f021c0abfc97" (UID: "e14af4c4-4ff7-4aa4-8586-f021c0abfc97"). InnerVolumeSpecName "kube-api-access-8q52f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:50:11.014643 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.014613 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q52f\" (UniqueName: \"kubernetes.io/projected/e14af4c4-4ff7-4aa4-8586-f021c0abfc97-kube-api-access-8q52f\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:50:11.612410 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.612375 4865 generic.go:356] "Generic (PLEG): container finished" podID="e14af4c4-4ff7-4aa4-8586-f021c0abfc97" containerID="eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449" exitCode=0 Oct 31 14:50:11.612842 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.612416 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" Oct 31 14:50:11.612842 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.612450 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" event={"ID":"e14af4c4-4ff7-4aa4-8586-f021c0abfc97","Type":"ContainerDied","Data":"eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449"} Oct 31 14:50:11.612842 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.612515 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lgtlw" event={"ID":"e14af4c4-4ff7-4aa4-8586-f021c0abfc97","Type":"ContainerDied","Data":"ac82464b0de5b90299f94826d8becec1c7ce62d9c717a7052fb57b72ea946239"} Oct 31 14:50:11.612842 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.612535 4865 scope.go:122] "RemoveContainer" containerID="eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449" Oct 31 14:50:11.621672 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.621648 4865 scope.go:122] "RemoveContainer" containerID="eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449" Oct 31 14:50:11.621988 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:50:11.621962 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449\": container with ID starting with eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449 not found: ID does not exist" containerID="eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449" Oct 31 14:50:11.622101 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.621998 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449"} err="failed to get container status \"eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449\": rpc error: code = NotFound desc = could not find container \"eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449\": container with ID starting with eb0a52d87ce227bf8b391201c68f4a38ceb40c0ef8cd56fee81f1fca6e841449 not found: ID does not exist" Oct 31 14:50:11.777505 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.777453 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lgtlw"] Oct 31 14:50:11.781090 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:11.781066 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lgtlw"] Oct 31 14:50:12.059531 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:12.059500 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:12.192918 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:12.192888 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:12.494045 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:12.494005 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14af4c4-4ff7-4aa4-8586-f021c0abfc97" path="/var/lib/kubelet/pods/e14af4c4-4ff7-4aa4-8586-f021c0abfc97/volumes" Oct 31 14:50:12.496077 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:12.496054 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:12.623876 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:12.623849 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:12.864558 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:12.864452 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:13.006665 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:13.006629 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:17.489260 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:50:17.489212 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:50:18.326584 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:18.326547 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:19.395929 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:19.395896 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:19.604248 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:19.604217 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2f7n9" Oct 31 14:50:23.084031 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:23.083992 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:23.233108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:23.233060 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:27.058549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:27.058516 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:27.188774 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:27.188746 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:27.818981 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:27.818936 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:45.886611 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:45.886562 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:47.489002 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:50:47.488968 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:50:48.832517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:48.832451 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:48.966674 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:48.966638 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:49.093335 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:49.093260 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:49.226773 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:49.226740 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:54.503915 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:54.503894 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 14:50:54.512194 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:54.512175 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 14:50:54.512726 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:54.512708 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:50:54.520523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:54.520509 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:50:56.010932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:56.010898 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:56.154607 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:56.154561 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:59.198388 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:59.198352 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:50:59.443634 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:50:59.443591 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:09.294174 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:09.294093 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:12.246007 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:12.245969 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:12.404229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:12.404191 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:12.892257 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:12.892222 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4"] Oct 31 14:51:12.892576 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:12.892565 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e14af4c4-4ff7-4aa4-8586-f021c0abfc97" containerName="authorino" Oct 31 14:51:12.892627 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:12.892578 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14af4c4-4ff7-4aa4-8586-f021c0abfc97" containerName="authorino" Oct 31 14:51:12.892662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:12.892654 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14af4c4-4ff7-4aa4-8586-f021c0abfc97" containerName="authorino" Oct 31 14:51:12.894283 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:12.894266 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.023973 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.023941 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4"] Oct 31 14:51:13.045323 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.045296 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.045459 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.045325 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.045459 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.045359 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af15b77c-21ac-4120-9571-9729dbb6585f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.045459 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.045407 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.045596 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.045487 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.045596 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.045545 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579rw\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-kube-api-access-579rw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.045596 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.045573 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.146498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.146408 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af15b77c-21ac-4120-9571-9729dbb6585f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.146498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.146443 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.146498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.146485 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.146764 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.146522 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579rw\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-kube-api-access-579rw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.146764 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.146546 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.146764 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.146574 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.146764 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.146601 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.147337 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.147262 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.148901 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.148876 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af15b77c-21ac-4120-9571-9729dbb6585f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.149095 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.149073 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.149136 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.149081 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.149136 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.149085 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.205526 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.205500 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579rw\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-kube-api-access-579rw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.206015 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.205993 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6n8v4\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.503585 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.503552 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:13.930078 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.925735 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4"] Oct 31 14:51:13.931999 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.931952 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:51:13.932098 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:13.932031 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:51:14.878679 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:14.878639 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" event={"ID":"af15b77c-21ac-4120-9571-9729dbb6585f","Type":"ContainerStarted","Data":"408eb18a9e2d127fd20e5fe042236ccb6ca5a91b8408d3633d806091c5cd3cec"} Oct 31 14:51:14.879178 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:14.878695 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" event={"ID":"af15b77c-21ac-4120-9571-9729dbb6585f","Type":"ContainerStarted","Data":"9e6acdae0c54e16162a69b73cd59bcf2f666d990ad80af3b6686ca914120205f"} Oct 31 14:51:14.879178 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:14.878821 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:14.880530 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:14.880503 4865 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6n8v4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:51:14.880655 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:14.880553 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" podUID="af15b77c-21ac-4120-9571-9729dbb6585f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:51:15.090017 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:15.089966 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" podStartSLOduration=3.089952978 podStartE2EDuration="3.089952978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:15.048083398 +0000 UTC m=+320.667634823" watchObservedRunningTime="2026-10-31 14:51:15.089952978 +0000 UTC m=+320.709504388" Oct 31 14:51:15.883710 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:15.883682 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 14:51:16.107169 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.107138 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922"] Oct 31 14:51:16.107339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.107323 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerName="discovery" containerID="cri-o://79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1" gracePeriod=30 Oct 31 14:51:16.138942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.138873 4865 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-68922 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/ready\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body= Oct 31 14:51:16.138942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.138921 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.25:8080/ready\": dial tcp 10.132.0.25:8080: connect: connection refused" Oct 31 14:51:16.352979 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.352957 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:51:16.474551 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.474524 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-kube-api-access-w6d9n\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-kube-api-access-w6d9n\") pod \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " Oct 31 14:51:16.474721 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.474564 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-token\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-token\") pod \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " Oct 31 14:51:16.474721 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.474617 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-cacerts\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-cacerts\") pod \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " Oct 31 14:51:16.474721 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.474643 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-dns-cert\") pod \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " Oct 31 14:51:16.474721 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.474676 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217ef3bf-f7b3-4eca-bb90-a45edb46be17-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/217ef3bf-f7b3-4eca-bb90-a45edb46be17-local-certs\") pod \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " Oct 31 14:51:16.474946 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.474743 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-ca-configmap\") pod \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " Oct 31 14:51:16.474946 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.474776 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-kubeconfig\") pod \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\" (UID: \"217ef3bf-f7b3-4eca-bb90-a45edb46be17\") " Oct 31 14:51:16.475422 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.475376 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-ca-configmap" pod "217ef3bf-f7b3-4eca-bb90-a45edb46be17" (UID: "217ef3bf-f7b3-4eca-bb90-a45edb46be17"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:51:16.477942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.477895 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-token" pod "217ef3bf-f7b3-4eca-bb90-a45edb46be17" (UID: "217ef3bf-f7b3-4eca-bb90-a45edb46be17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:51:16.478145 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.478103 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-cacerts" pod "217ef3bf-f7b3-4eca-bb90-a45edb46be17" (UID: "217ef3bf-f7b3-4eca-bb90-a45edb46be17"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:51:16.478354 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.478310 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-kube-api-access-w6d9n" pod "217ef3bf-f7b3-4eca-bb90-a45edb46be17" (UID: "217ef3bf-f7b3-4eca-bb90-a45edb46be17"). InnerVolumeSpecName "kube-api-access-w6d9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:51:16.479708 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.479681 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-kubeconfig" pod "217ef3bf-f7b3-4eca-bb90-a45edb46be17" (UID: "217ef3bf-f7b3-4eca-bb90-a45edb46be17"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:51:16.479803 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.479762 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-dns-cert" pod "217ef3bf-f7b3-4eca-bb90-a45edb46be17" (UID: "217ef3bf-f7b3-4eca-bb90-a45edb46be17"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:51:16.479877 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.479857 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217ef3bf-f7b3-4eca-bb90-a45edb46be17-local-certs" pod "217ef3bf-f7b3-4eca-bb90-a45edb46be17" (UID: "217ef3bf-f7b3-4eca-bb90-a45edb46be17"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:51:16.575794 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.575761 4865 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-ca-configmap\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:16.575794 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.575794 4865 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-kubeconfig\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:16.575794 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.575804 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6d9n\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-kube-api-access-w6d9n\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:16.576068 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.575814 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:16.576068 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.575823 4865 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-cacerts\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:16.576068 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.575832 4865 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/217ef3bf-f7b3-4eca-bb90-a45edb46be17-istio-csr-dns-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:16.576068 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.575840 4865 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/217ef3bf-f7b3-4eca-bb90-a45edb46be17-local-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:16.888727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.888645 4865 generic.go:356] "Generic (PLEG): container finished" podID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerID="79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1" exitCode=0 Oct 31 14:51:16.888727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.888705 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" Oct 31 14:51:16.889197 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.888728 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" event={"ID":"217ef3bf-f7b3-4eca-bb90-a45edb46be17","Type":"ContainerDied","Data":"79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1"} Oct 31 14:51:16.889197 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.888777 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922" event={"ID":"217ef3bf-f7b3-4eca-bb90-a45edb46be17","Type":"ContainerDied","Data":"c4ca99d0d7563a24c7e20281869d06b6fad05c0d78651c37e518608d9e2b2cca"} Oct 31 14:51:16.889197 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.888798 4865 scope.go:122] "RemoveContainer" containerID="79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1" Oct 31 14:51:16.897357 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.897328 4865 scope.go:122] "RemoveContainer" containerID="79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1" Oct 31 14:51:16.897730 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:51:16.897709 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1\": container with ID starting with 79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1 not found: ID does not exist" containerID="79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1" Oct 31 14:51:16.897801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:16.897738 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1"} err="failed to get container status \"79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1\": rpc error: code = NotFound desc = could not find container \"79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1\": container with ID starting with 79f0f2b27d62cfc36f6ae0741b10d3bbb9ea6b4144704798ab09e96f6811e4b1 not found: ID does not exist" Oct 31 14:51:17.055912 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:17.055884 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922"] Oct 31 14:51:17.106822 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:17.106793 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-68922"] Oct 31 14:51:17.489801 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:51:17.489769 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:51:18.122711 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.122677 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:18.247781 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.247745 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:18.370784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.370748 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:18.494706 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.494675 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" path="/var/lib/kubelet/pods/217ef3bf-f7b3-4eca-bb90-a45edb46be17/volumes" Oct 31 14:51:18.504130 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.502851 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:18.630141 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.630106 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:18.763319 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.763243 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:18.897049 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.897013 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:18.992149 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.992118 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lz97q"] Oct 31 14:51:18.992433 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.992421 4865 cpu_manager.go:414] "containerMap: removing container" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerName="discovery" Oct 31 14:51:18.992512 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.992435 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerName="discovery" Oct 31 14:51:18.992560 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.992528 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="217ef3bf-f7b3-4eca-bb90-a45edb46be17" containerName="discovery" Oct 31 14:51:18.996762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:18.996744 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.023769 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.023696 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:19.050446 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.050384 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 14:51:19.097644 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.097603 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dcl\" (UniqueName: \"kubernetes.io/projected/200047c9-fccd-4826-b521-fce830acd38f-kube-api-access-l8dcl\") pod \"kserve-controller-manager-776cc66d47-lz97q\" (UID: \"200047c9-fccd-4826-b521-fce830acd38f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.097644 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.097650 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/200047c9-fccd-4826-b521-fce830acd38f-cert\") pod \"kserve-controller-manager-776cc66d47-lz97q\" (UID: \"200047c9-fccd-4826-b521-fce830acd38f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.101352 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.101316 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 14:51:19.101352 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.101323 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8njhh\"" Oct 31 14:51:19.101550 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.101348 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 14:51:19.102610 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.102594 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lz97q"] Oct 31 14:51:19.176877 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.176844 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:19.199036 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.199003 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dcl\" (UniqueName: \"kubernetes.io/projected/200047c9-fccd-4826-b521-fce830acd38f-kube-api-access-l8dcl\") pod \"kserve-controller-manager-776cc66d47-lz97q\" (UID: \"200047c9-fccd-4826-b521-fce830acd38f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.199194 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.199041 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/200047c9-fccd-4826-b521-fce830acd38f-cert\") pod \"kserve-controller-manager-776cc66d47-lz97q\" (UID: \"200047c9-fccd-4826-b521-fce830acd38f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.201709 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.201687 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/200047c9-fccd-4826-b521-fce830acd38f-cert\") pod \"kserve-controller-manager-776cc66d47-lz97q\" (UID: \"200047c9-fccd-4826-b521-fce830acd38f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.257321 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.257290 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dcl\" (UniqueName: \"kubernetes.io/projected/200047c9-fccd-4826-b521-fce830acd38f-kube-api-access-l8dcl\") pod \"kserve-controller-manager-776cc66d47-lz97q\" (UID: \"200047c9-fccd-4826-b521-fce830acd38f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.309389 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.309310 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:19.327544 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.327520 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5vzk"] Oct 31 14:51:19.331043 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.331023 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.388955 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.388927 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 14:51:19.389127 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.389112 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zxl7k\"" Oct 31 14:51:19.389181 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.389169 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 14:51:19.390165 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.390144 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5vzk"] Oct 31 14:51:19.401688 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.401661 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrdm\" (UniqueName: \"kubernetes.io/projected/4e96a5f7-706c-4405-be53-e4362dc7677d-kube-api-access-jkrdm\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.401824 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.401698 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e96a5f7-706c-4405-be53-e4362dc7677d-cert\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.401824 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.401717 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e96a5f7-706c-4405-be53-e4362dc7677d-metrics-certs\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.502265 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.502228 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrdm\" (UniqueName: \"kubernetes.io/projected/4e96a5f7-706c-4405-be53-e4362dc7677d-kube-api-access-jkrdm\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.502449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.502273 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e96a5f7-706c-4405-be53-e4362dc7677d-cert\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.502449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.502326 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e96a5f7-706c-4405-be53-e4362dc7677d-metrics-certs\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.505017 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.504987 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e96a5f7-706c-4405-be53-e4362dc7677d-cert\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.505017 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.505012 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e96a5f7-706c-4405-be53-e4362dc7677d-metrics-certs\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.564043 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.563977 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrdm\" (UniqueName: \"kubernetes.io/projected/4e96a5f7-706c-4405-be53-e4362dc7677d-kube-api-access-jkrdm\") pod \"odh-model-controller-769875db56-n5vzk\" (UID: \"4e96a5f7-706c-4405-be53-e4362dc7677d\") " pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.644200 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.644168 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:19.685693 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.685663 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lz97q"] Oct 31 14:51:19.687926 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:51:19.687900 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200047c9_fccd_4826_b521_fce830acd38f.slice/crio-13cdede4def4c5297cc9ae67df9e23d581b284974c19e43c809ba1f0610d8887 WatchSource:0}: Error finding container 13cdede4def4c5297cc9ae67df9e23d581b284974c19e43c809ba1f0610d8887: Status 404 returned error can't find the container with id 13cdede4def4c5297cc9ae67df9e23d581b284974c19e43c809ba1f0610d8887 Oct 31 14:51:19.689773 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.689751 4865 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:51:19.905376 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:19.905282 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" event={"ID":"200047c9-fccd-4826-b521-fce830acd38f","Type":"ContainerStarted","Data":"13cdede4def4c5297cc9ae67df9e23d581b284974c19e43c809ba1f0610d8887"} Oct 31 14:51:20.012749 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:20.012726 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5vzk"] Oct 31 14:51:20.014653 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:51:20.014630 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e96a5f7_706c_4405_be53_e4362dc7677d.slice/crio-0ad23e4a43eec6a70994e5b60cb20ef9eae724fcfdc37998f8b236d2a1b3ec86 WatchSource:0}: Error finding container 0ad23e4a43eec6a70994e5b60cb20ef9eae724fcfdc37998f8b236d2a1b3ec86: Status 404 returned error can't find the container with id 0ad23e4a43eec6a70994e5b60cb20ef9eae724fcfdc37998f8b236d2a1b3ec86 Oct 31 14:51:20.365963 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:20.365933 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:20.911310 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:20.911273 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n5vzk" event={"ID":"4e96a5f7-706c-4405-be53-e4362dc7677d","Type":"ContainerStarted","Data":"0ad23e4a43eec6a70994e5b60cb20ef9eae724fcfdc37998f8b236d2a1b3ec86"} Oct 31 14:51:22.925180 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:22.925139 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" event={"ID":"200047c9-fccd-4826-b521-fce830acd38f","Type":"ContainerStarted","Data":"a9c14908b66183855f1592c0502317c45189d8f18410711c4a58636937a02b51"} Oct 31 14:51:22.925709 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:22.925300 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:51:23.042247 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:23.042176 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" podStartSLOduration=2.447326968 podStartE2EDuration="5.042158956s" totalImagesPullingTime="2.594831988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:23.040590207 +0000 UTC m=+328.660141622" watchObservedRunningTime="2026-10-31 14:51:23.042158956 +0000 UTC m=+328.661710363" Oct 31 14:51:23.932162 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:23.932125 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n5vzk" event={"ID":"4e96a5f7-706c-4405-be53-e4362dc7677d","Type":"ContainerStarted","Data":"531e05ce4a9b6de310e40fb615122df540d323789992b37d679330ed527401b2"} Oct 31 14:51:24.048554 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:24.048507 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-n5vzk" podStartSLOduration=1.937705447 podStartE2EDuration="5.048463994s" totalImagesPullingTime="3.110758547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:24.047484619 +0000 UTC m=+329.667036024" watchObservedRunningTime="2026-10-31 14:51:24.048463994 +0000 UTC m=+329.668015404" Oct 31 14:51:24.936230 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:24.936200 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:25.369559 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:25.369527 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:25.499203 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:25.499171 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:25.628950 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:25.628876 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:25.760735 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:25.760701 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:25.884839 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:25.884768 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:26.005930 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:26.005897 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:26.144067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:26.143975 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:26.265493 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:26.265443 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:26.385462 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:26.385432 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:26.509031 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:26.508998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:29.369988 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:29.369956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:29.505932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:29.505896 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:31.576402 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:31.576370 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:31.741009 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:31.740978 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:31.867541 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:31.867454 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:32.038027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:32.037995 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:35.942740 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:35.942712 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-n5vzk" Oct 31 14:51:37.995894 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:37.995857 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:51:39.475355 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:39.475319 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:39.606855 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:39.606817 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:46.237089 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:46.237057 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:46.360654 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:46.360621 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:47.490542 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:51:47.490502 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:51:47.897577 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:47.897540 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:48.169934 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.169840 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qz79l"] Oct 31 14:51:48.174048 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.174023 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" Oct 31 14:51:48.177571 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.177547 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:51:48.227408 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.227380 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lr8w6\"" Oct 31 14:51:48.243483 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.243429 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7x7\" (UniqueName: \"kubernetes.io/projected/e5359a5d-97b1-44a8-8a97-c381aa409669-kube-api-access-4z7x7\") pod \"authorino-6f6944bcdd-qz79l\" (UID: \"e5359a5d-97b1-44a8-8a97-c381aa409669\") " pod="kuadrant-system/authorino-6f6944bcdd-qz79l" Oct 31 14:51:48.279504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.279464 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qz79l"] Oct 31 14:51:48.344005 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.343981 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7x7\" (UniqueName: \"kubernetes.io/projected/e5359a5d-97b1-44a8-8a97-c381aa409669-kube-api-access-4z7x7\") pod \"authorino-6f6944bcdd-qz79l\" (UID: \"e5359a5d-97b1-44a8-8a97-c381aa409669\") " pod="kuadrant-system/authorino-6f6944bcdd-qz79l" Oct 31 14:51:48.401577 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.401545 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7x7\" (UniqueName: \"kubernetes.io/projected/e5359a5d-97b1-44a8-8a97-c381aa409669-kube-api-access-4z7x7\") pod \"authorino-6f6944bcdd-qz79l\" (UID: \"e5359a5d-97b1-44a8-8a97-c381aa409669\") " pod="kuadrant-system/authorino-6f6944bcdd-qz79l" Oct 31 14:51:48.500147 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.500122 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" Oct 31 14:51:48.512160 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.512135 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qz79l"] Oct 31 14:51:48.537422 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.537394 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c9c55bc97-w7qfc"] Oct 31 14:51:48.541948 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.541922 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" Oct 31 14:51:48.546290 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.546264 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf96r\" (UniqueName: \"kubernetes.io/projected/b327f2c3-2ea0-4e90-8209-1575e3eb1107-kube-api-access-zf96r\") pod \"authorino-5c9c55bc97-w7qfc\" (UID: \"b327f2c3-2ea0-4e90-8209-1575e3eb1107\") " pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" Oct 31 14:51:48.593748 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.593720 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c9c55bc97-w7qfc"] Oct 31 14:51:48.647409 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.647380 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf96r\" (UniqueName: \"kubernetes.io/projected/b327f2c3-2ea0-4e90-8209-1575e3eb1107-kube-api-access-zf96r\") pod \"authorino-5c9c55bc97-w7qfc\" (UID: \"b327f2c3-2ea0-4e90-8209-1575e3eb1107\") " pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" Oct 31 14:51:48.705389 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.705358 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf96r\" (UniqueName: \"kubernetes.io/projected/b327f2c3-2ea0-4e90-8209-1575e3eb1107-kube-api-access-zf96r\") pod \"authorino-5c9c55bc97-w7qfc\" (UID: \"b327f2c3-2ea0-4e90-8209-1575e3eb1107\") " pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" Oct 31 14:51:48.855520 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.855426 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" Oct 31 14:51:48.867946 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.867924 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qz79l"] Oct 31 14:51:48.870917 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:51:48.870889 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5359a5d_97b1_44a8_8a97_c381aa409669.slice/crio-ce9689670e8673b06efbbd93674aff23117f6b6c8c70afc79fad87e9632d17a1 WatchSource:0}: Error finding container ce9689670e8673b06efbbd93674aff23117f6b6c8c70afc79fad87e9632d17a1: Status 404 returned error can't find the container with id ce9689670e8673b06efbbd93674aff23117f6b6c8c70afc79fad87e9632d17a1 Oct 31 14:51:48.993532 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:48.993501 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c9c55bc97-w7qfc"] Oct 31 14:51:49.020630 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.020599 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-596f45b59f-4t67l"] Oct 31 14:51:49.025001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.024978 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.040768 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.040739 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" event={"ID":"e5359a5d-97b1-44a8-8a97-c381aa409669","Type":"ContainerStarted","Data":"ce9689670e8673b06efbbd93674aff23117f6b6c8c70afc79fad87e9632d17a1"} Oct 31 14:51:49.051160 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.051126 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdsqd\" (UniqueName: \"kubernetes.io/projected/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-kube-api-access-zdsqd\") pod \"authorino-596f45b59f-4t67l\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.051267 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.051176 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-tls-cert\") pod \"authorino-596f45b59f-4t67l\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.078398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.078377 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-596f45b59f-4t67l"] Oct 31 14:51:49.080828 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.080811 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 14:51:49.152438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.152362 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdsqd\" (UniqueName: \"kubernetes.io/projected/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-kube-api-access-zdsqd\") pod \"authorino-596f45b59f-4t67l\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.152438 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.152406 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-tls-cert\") pod \"authorino-596f45b59f-4t67l\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.155009 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.154984 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-tls-cert\") pod \"authorino-596f45b59f-4t67l\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.210315 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.210283 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdsqd\" (UniqueName: \"kubernetes.io/projected/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-kube-api-access-zdsqd\") pod \"authorino-596f45b59f-4t67l\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.222004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.221967 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c9c55bc97-w7qfc"] Oct 31 14:51:49.224798 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:51:49.224773 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb327f2c3_2ea0_4e90_8209_1575e3eb1107.slice/crio-7e5759f036013c52a9da9bcfd3eefc4caf5c7d9c3e8313fc882332f3a56e833f WatchSource:0}: Error finding container 7e5759f036013c52a9da9bcfd3eefc4caf5c7d9c3e8313fc882332f3a56e833f: Status 404 returned error can't find the container with id 7e5759f036013c52a9da9bcfd3eefc4caf5c7d9c3e8313fc882332f3a56e833f Oct 31 14:51:49.336871 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.336842 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:51:49.705625 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:49.705599 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-596f45b59f-4t67l"] Oct 31 14:51:49.707275 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:51:49.707234 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d1a08f8_fb63_45b6_8fd0_c0b875f54e6e.slice/crio-79eea8d246e34b6b33e18515a3ed10706c97e4dd39f4821fbdf3770ea2524ede WatchSource:0}: Error finding container 79eea8d246e34b6b33e18515a3ed10706c97e4dd39f4821fbdf3770ea2524ede: Status 404 returned error can't find the container with id 79eea8d246e34b6b33e18515a3ed10706c97e4dd39f4821fbdf3770ea2524ede Oct 31 14:51:50.045989 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:50.045946 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" event={"ID":"b327f2c3-2ea0-4e90-8209-1575e3eb1107","Type":"ContainerStarted","Data":"7e5759f036013c52a9da9bcfd3eefc4caf5c7d9c3e8313fc882332f3a56e833f"} Oct 31 14:51:50.047091 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:50.047068 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-596f45b59f-4t67l" event={"ID":"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e","Type":"ContainerStarted","Data":"79eea8d246e34b6b33e18515a3ed10706c97e4dd39f4821fbdf3770ea2524ede"} Oct 31 14:51:51.056094 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.056043 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" event={"ID":"e5359a5d-97b1-44a8-8a97-c381aa409669","Type":"ContainerStarted","Data":"7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338"} Oct 31 14:51:51.056590 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.056258 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" podUID="e5359a5d-97b1-44a8-8a97-c381aa409669" containerName="authorino" containerID="cri-o://7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338" gracePeriod=30 Oct 31 14:51:51.058682 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.058652 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" event={"ID":"b327f2c3-2ea0-4e90-8209-1575e3eb1107","Type":"ContainerStarted","Data":"69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354"} Oct 31 14:51:51.058822 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.058797 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" podUID="b327f2c3-2ea0-4e90-8209-1575e3eb1107" containerName="authorino" containerID="cri-o://69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354" gracePeriod=30 Oct 31 14:51:51.061350 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.061320 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-596f45b59f-4t67l" event={"ID":"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e","Type":"ContainerStarted","Data":"3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6"} Oct 31 14:51:51.174361 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.174306 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" podStartSLOduration=1.6804752760000001 podStartE2EDuration="3.174287913s" totalImagesPullingTime="1.493812637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:51.173987157 +0000 UTC m=+356.793538569" watchObservedRunningTime="2026-10-31 14:51:51.174287913 +0000 UTC m=+356.793839327" Oct 31 14:51:51.289457 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.289405 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" podStartSLOduration=2.057335109 podStartE2EDuration="3.289387794s" totalImagesPullingTime="1.232052685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:51.288281285 +0000 UTC m=+356.907832697" watchObservedRunningTime="2026-10-31 14:51:51.289387794 +0000 UTC m=+356.908939204" Oct 31 14:51:51.313171 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.313147 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" Oct 31 14:51:51.321767 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.321747 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" Oct 31 14:51:51.368638 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.368609 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b327f2c3-2ea0-4e90-8209-1575e3eb1107-kube-api-access-zf96r\" (UniqueName: \"kubernetes.io/projected/b327f2c3-2ea0-4e90-8209-1575e3eb1107-kube-api-access-zf96r\") pod \"b327f2c3-2ea0-4e90-8209-1575e3eb1107\" (UID: \"b327f2c3-2ea0-4e90-8209-1575e3eb1107\") " Oct 31 14:51:51.368752 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.368723 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5359a5d-97b1-44a8-8a97-c381aa409669-kube-api-access-4z7x7\" (UniqueName: \"kubernetes.io/projected/e5359a5d-97b1-44a8-8a97-c381aa409669-kube-api-access-4z7x7\") pod \"e5359a5d-97b1-44a8-8a97-c381aa409669\" (UID: \"e5359a5d-97b1-44a8-8a97-c381aa409669\") " Oct 31 14:51:51.370793 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.370769 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5359a5d-97b1-44a8-8a97-c381aa409669-kube-api-access-4z7x7" pod "e5359a5d-97b1-44a8-8a97-c381aa409669" (UID: "e5359a5d-97b1-44a8-8a97-c381aa409669"). InnerVolumeSpecName "kube-api-access-4z7x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:51:51.370868 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.370788 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327f2c3-2ea0-4e90-8209-1575e3eb1107-kube-api-access-zf96r" pod "b327f2c3-2ea0-4e90-8209-1575e3eb1107" (UID: "b327f2c3-2ea0-4e90-8209-1575e3eb1107"). InnerVolumeSpecName "kube-api-access-zf96r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:51:51.405024 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.404981 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-596f45b59f-4t67l" podStartSLOduration=2.217322497 podStartE2EDuration="3.404966357s" totalImagesPullingTime="1.18764386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:51.403360634 +0000 UTC m=+357.022912044" watchObservedRunningTime="2026-10-31 14:51:51.404966357 +0000 UTC m=+357.024517768" Oct 31 14:51:51.469875 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.469844 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z7x7\" (UniqueName: \"kubernetes.io/projected/e5359a5d-97b1-44a8-8a97-c381aa409669-kube-api-access-4z7x7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:51.470024 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:51.469879 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf96r\" (UniqueName: \"kubernetes.io/projected/b327f2c3-2ea0-4e90-8209-1575e3eb1107-kube-api-access-zf96r\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:52.068233 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.068197 4865 generic.go:356] "Generic (PLEG): container finished" podID="e5359a5d-97b1-44a8-8a97-c381aa409669" containerID="7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338" exitCode=0 Oct 31 14:51:52.068691 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.068260 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" Oct 31 14:51:52.068691 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.068282 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" event={"ID":"e5359a5d-97b1-44a8-8a97-c381aa409669","Type":"ContainerDied","Data":"7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338"} Oct 31 14:51:52.068691 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.068319 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-qz79l" event={"ID":"e5359a5d-97b1-44a8-8a97-c381aa409669","Type":"ContainerDied","Data":"ce9689670e8673b06efbbd93674aff23117f6b6c8c70afc79fad87e9632d17a1"} Oct 31 14:51:52.068691 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.068343 4865 scope.go:122] "RemoveContainer" containerID="7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338" Oct 31 14:51:52.069743 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.069609 4865 generic.go:356] "Generic (PLEG): container finished" podID="b327f2c3-2ea0-4e90-8209-1575e3eb1107" containerID="69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354" exitCode=0 Oct 31 14:51:52.069743 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.069660 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" Oct 31 14:51:52.069743 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.069715 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" event={"ID":"b327f2c3-2ea0-4e90-8209-1575e3eb1107","Type":"ContainerDied","Data":"69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354"} Oct 31 14:51:52.069886 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.069754 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9c55bc97-w7qfc" event={"ID":"b327f2c3-2ea0-4e90-8209-1575e3eb1107","Type":"ContainerDied","Data":"7e5759f036013c52a9da9bcfd3eefc4caf5c7d9c3e8313fc882332f3a56e833f"} Oct 31 14:51:52.079384 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.079367 4865 scope.go:122] "RemoveContainer" containerID="7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338" Oct 31 14:51:52.079744 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:51:52.079691 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338\": container with ID starting with 7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338 not found: ID does not exist" containerID="7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338" Oct 31 14:51:52.079744 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.079718 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338"} err="failed to get container status \"7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338\": rpc error: code = NotFound desc = could not find container \"7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338\": container with ID starting with 7f5ade094f25ae6ae0c4eb741badca26239545ed871848c45b0985e2dbba5338 not found: ID does not exist" Oct 31 14:51:52.079744 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.079735 4865 scope.go:122] "RemoveContainer" containerID="69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354" Oct 31 14:51:52.087604 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.087585 4865 scope.go:122] "RemoveContainer" containerID="69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354" Oct 31 14:51:52.087916 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:51:52.087894 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354\": container with ID starting with 69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354 not found: ID does not exist" containerID="69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354" Oct 31 14:51:52.087998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.087920 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354"} err="failed to get container status \"69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354\": rpc error: code = NotFound desc = could not find container \"69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354\": container with ID starting with 69875504a877ce0a64c8367e0ea11697472cc87587df40075b5f1caa2f73a354 not found: ID does not exist" Oct 31 14:51:52.234457 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.234422 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c9c55bc97-w7qfc"] Oct 31 14:51:52.242543 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.242519 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c9c55bc97-w7qfc"] Oct 31 14:51:52.352686 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.352616 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qz79l"] Oct 31 14:51:52.356632 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.356605 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-qz79l"] Oct 31 14:51:52.494860 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.494827 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b327f2c3-2ea0-4e90-8209-1575e3eb1107" path="/var/lib/kubelet/pods/b327f2c3-2ea0-4e90-8209-1575e3eb1107/volumes" Oct 31 14:51:52.495182 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:52.495169 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5359a5d-97b1-44a8-8a97-c381aa409669" path="/var/lib/kubelet/pods/e5359a5d-97b1-44a8-8a97-c381aa409669/volumes" Oct 31 14:51:53.937899 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:51:53.937874 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lz97q" Oct 31 14:52:17.489567 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:52:17.489532 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:52:22.788290 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:52:22.788216 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:52:22.929003 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:52:22.928966 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:52:46.581883 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:52:46.581850 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:52:47.489081 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:52:47.489047 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:52:56.655104 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:52:56.655067 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:08.494120 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:08.494088 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:08.634976 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:08.634938 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:14.437748 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:14.437710 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:14.613724 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:14.613694 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:17.488941 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:53:17.488905 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:53:38.564045 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:38.564015 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:39.083445 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:39.083407 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:40.308296 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:40.308265 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:40.453443 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:40.453406 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:41.447973 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:41.447936 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:41.631798 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:41.631767 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:46.528505 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:46.528445 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:46.914782 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:46.914714 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:47.488942 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:53:47.488900 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:53:50.398255 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:50.398223 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:53:52.806147 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.806111 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j"] Oct 31 14:53:52.806573 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.806458 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b327f2c3-2ea0-4e90-8209-1575e3eb1107" containerName="authorino" Oct 31 14:53:52.806573 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.806483 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b327f2c3-2ea0-4e90-8209-1575e3eb1107" containerName="authorino" Oct 31 14:53:52.806573 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.806493 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e5359a5d-97b1-44a8-8a97-c381aa409669" containerName="authorino" Oct 31 14:53:52.806573 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.806502 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5359a5d-97b1-44a8-8a97-c381aa409669" containerName="authorino" Oct 31 14:53:52.806573 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.806571 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b327f2c3-2ea0-4e90-8209-1575e3eb1107" containerName="authorino" Oct 31 14:53:52.806744 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.806579 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5359a5d-97b1-44a8-8a97-c381aa409669" containerName="authorino" Oct 31 14:53:52.809667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.809652 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.865744 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.865692 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.865744 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.865754 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.866378 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.866353 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.866595 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.866575 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.866803 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.866783 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvhx\" (UniqueName: \"kubernetes.io/projected/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-kube-api-access-4xvhx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.866893 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.866857 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.868017 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.867997 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j"] Oct 31 14:53:52.887153 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.887124 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:52.916941 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.916920 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 14:53:52.917180 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.917161 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 14:53:52.917252 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.917179 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 14:53:52.917252 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.917226 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m92fm\"" Oct 31 14:53:52.968217 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968190 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.968350 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968224 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.968350 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968241 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvhx\" (UniqueName: \"kubernetes.io/projected/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-kube-api-access-4xvhx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.968452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968354 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.968452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968392 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.968452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968417 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.968879 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968852 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.968879 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968868 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.969187 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.968876 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.970632 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.970612 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:52.971008 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:52.970989 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:53.026118 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:53.026088 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvhx\" (UniqueName: \"kubernetes.io/projected/4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8-kube-api-access-4xvhx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22b9j\" (UID: \"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:53.122343 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:53.122244 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:53.493878 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:53.493855 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j"] Oct 31 14:53:53.495268 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:53:53.495239 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ded9cdd_cf9d_4bf8_a6aa_e519619b8ea8.slice/crio-c4f2e6a818f839bdb6b5682b170994d6f6cfe4af412ac86b43e1b8f402f97edc WatchSource:0}: Error finding container c4f2e6a818f839bdb6b5682b170994d6f6cfe4af412ac86b43e1b8f402f97edc: Status 404 returned error can't find the container with id c4f2e6a818f839bdb6b5682b170994d6f6cfe4af412ac86b43e1b8f402f97edc Oct 31 14:53:53.586791 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:53.586763 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" event={"ID":"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8","Type":"ContainerStarted","Data":"c4f2e6a818f839bdb6b5682b170994d6f6cfe4af412ac86b43e1b8f402f97edc"} Oct 31 14:53:55.597728 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:55.597683 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" event={"ID":"4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8","Type":"ContainerStarted","Data":"fffa94c23d21dc096838d4271f99a73e96bd98ce01facb7ae50d17fd24db5e7f"} Oct 31 14:53:55.723639 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:55.723575 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" podStartSLOduration=2.142746663 podStartE2EDuration="3.723561849s" totalImagesPullingTime="1.580815186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:53:55.722273775 +0000 UTC m=+481.341825177" watchObservedRunningTime="2026-10-31 14:53:55.723561849 +0000 UTC m=+481.343113251" Oct 31 14:53:55.954376 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:55.954274 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:56.122512 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:56.122453 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:53:58.996341 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:58.996309 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8"] Oct 31 14:53:58.999821 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:58.999804 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.090427 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.090397 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:53:59.105977 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.105957 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 14:53:59.107414 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.107395 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8"] Oct 31 14:53:59.132457 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.132434 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8fa3fc-a649-4b87-b616-4156319382e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.132542 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.132485 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.132542 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.132506 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbthw\" (UniqueName: \"kubernetes.io/projected/7b8fa3fc-a649-4b87-b616-4156319382e3-kube-api-access-kbthw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.132627 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.132600 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.132667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.132658 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.132719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.132701 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.233791 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.233758 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.233791 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.233792 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.234001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.233824 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8fa3fc-a649-4b87-b616-4156319382e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.234041 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.234019 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.234075 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.234061 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbthw\" (UniqueName: \"kubernetes.io/projected/7b8fa3fc-a649-4b87-b616-4156319382e3-kube-api-access-kbthw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.234129 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.234113 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.234195 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.234164 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.234256 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.234221 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.234383 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.234355 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.236247 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.236224 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b8fa3fc-a649-4b87-b616-4156319382e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.236564 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.236547 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8fa3fc-a649-4b87-b616-4156319382e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.292212 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.292140 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbthw\" (UniqueName: \"kubernetes.io/projected/7b8fa3fc-a649-4b87-b616-4156319382e3-kube-api-access-kbthw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r6fn8\" (UID: \"7b8fa3fc-a649-4b87-b616-4156319382e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.311996 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.311973 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:53:59.681602 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:53:59.681577 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8"] Oct 31 14:53:59.683515 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:53:59.683487 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8fa3fc_a649_4b87_b616_4156319382e3.slice/crio-83cef76a94970e5beac162fbc6a5b72adc8eced7675c3bcf8ad3ffe100507993 WatchSource:0}: Error finding container 83cef76a94970e5beac162fbc6a5b72adc8eced7675c3bcf8ad3ffe100507993: Status 404 returned error can't find the container with id 83cef76a94970e5beac162fbc6a5b72adc8eced7675c3bcf8ad3ffe100507993 Oct 31 14:54:00.620864 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:00.620787 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" event={"ID":"7b8fa3fc-a649-4b87-b616-4156319382e3","Type":"ContainerStarted","Data":"bf66f57eb5495d3e5b65fd507eb9df29da0c36a3db3696e45f049c43320a091a"} Oct 31 14:54:00.620864 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:00.620837 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" event={"ID":"7b8fa3fc-a649-4b87-b616-4156319382e3","Type":"ContainerStarted","Data":"83cef76a94970e5beac162fbc6a5b72adc8eced7675c3bcf8ad3ffe100507993"} Oct 31 14:54:00.740358 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:00.740276 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" podStartSLOduration=2.134327731 podStartE2EDuration="2.74025685s" totalImagesPullingTime="605.929119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:00.738136071 +0000 UTC m=+486.357687478" watchObservedRunningTime="2026-10-31 14:54:00.74025685 +0000 UTC m=+486.359808262" Oct 31 14:54:01.312916 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:01.312888 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:54:02.640480 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:02.640428 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:54:03.029662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:03.029629 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:03.123409 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:03.123372 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:54:03.141794 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:03.141769 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:54:03.645070 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:03.645042 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22b9j" Oct 31 14:54:06.624164 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.624090 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2fbkj"] Oct 31 14:54:06.627306 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.627288 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.701079 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.701047 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxl65\" (UniqueName: \"kubernetes.io/projected/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-kube-api-access-qxl65\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.701254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.701097 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.701254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.701118 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.701254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.701219 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.701368 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.701262 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-home\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.701368 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.701279 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.713738 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.713687 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:06.733717 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.733696 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 14:54:06.735091 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.735073 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2fbkj"] Oct 31 14:54:06.802635 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.802599 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-home\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.802635 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.802642 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.802881 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.802692 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxl65\" (UniqueName: \"kubernetes.io/projected/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-kube-api-access-qxl65\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.802881 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.802728 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.802881 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.802757 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.803052 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.802896 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.803153 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.803130 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-home\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.803205 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.803157 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.803243 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.803205 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.804990 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.804969 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.805385 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.805362 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.860906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.860878 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxl65\" (UniqueName: \"kubernetes.io/projected/1fd003d5-b7f4-42bf-87ef-bfa1a035d906-kube-api-access-qxl65\") pod \"echo-simulator-kserve-64b6f8f96c-2fbkj\" (UID: \"1fd003d5-b7f4-42bf-87ef-bfa1a035d906\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:06.939885 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:06.939798 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:07.315689 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.315664 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2fbkj"] Oct 31 14:54:07.317057 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:54:07.317030 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd003d5_b7f4_42bf_87ef_bfa1a035d906.slice/crio-83fe01ee6172dd0dd2a9e3034dbf48d077a5a31347488d75829919a1cc4ac2fb WatchSource:0}: Error finding container 83fe01ee6172dd0dd2a9e3034dbf48d077a5a31347488d75829919a1cc4ac2fb: Status 404 returned error can't find the container with id 83fe01ee6172dd0dd2a9e3034dbf48d077a5a31347488d75829919a1cc4ac2fb Oct 31 14:54:07.593359 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.593275 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln"] Oct 31 14:54:07.597954 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.597932 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.651075 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.651044 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" Oct 31 14:54:07.652187 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.652165 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" event={"ID":"1fd003d5-b7f4-42bf-87ef-bfa1a035d906","Type":"ContainerStarted","Data":"83fe01ee6172dd0dd2a9e3034dbf48d077a5a31347488d75829919a1cc4ac2fb"} Oct 31 14:54:07.703709 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.703679 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln"] Oct 31 14:54:07.711456 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711429 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8g9\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-kube-api-access-zc8g9\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711612 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711501 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711612 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711600 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711689 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711624 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711723 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711701 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711757 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711734 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711790 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711760 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711838 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711822 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.711910 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.711896 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813127 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813089 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813144 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813176 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813202 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813236 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813266 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8g9\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-kube-api-access-zc8g9\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813620 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813311 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813620 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813363 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813620 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813388 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813620 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813527 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813620 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813542 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813890 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813621 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813890 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813750 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.813890 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.813885 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.815880 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.815857 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.816117 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.816098 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.875115 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.875052 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.875419 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.875391 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8g9\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-kube-api-access-zc8g9\") pod \"maas-default-gateway-openshift-default-67bf986f99-rj2ln\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:07.910832 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:07.910799 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:08.350663 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:08.350629 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln"] Oct 31 14:54:08.373958 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:54:08.373926 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2c6faf_2b9a_4ee8_a34f_5118f1ba872b.slice/crio-74abe1fa8cf63754ed387feeeacd06c6dcb2720f2f7426b09f05b41e7a141aaf WatchSource:0}: Error finding container 74abe1fa8cf63754ed387feeeacd06c6dcb2720f2f7426b09f05b41e7a141aaf: Status 404 returned error can't find the container with id 74abe1fa8cf63754ed387feeeacd06c6dcb2720f2f7426b09f05b41e7a141aaf Oct 31 14:54:08.659258 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:08.659180 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" event={"ID":"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b","Type":"ContainerStarted","Data":"74abe1fa8cf63754ed387feeeacd06c6dcb2720f2f7426b09f05b41e7a141aaf"} Oct 31 14:54:09.313155 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:09.313128 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:54:09.331311 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:09.331280 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:54:09.680533 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:09.680433 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r6fn8" Oct 31 14:54:10.675547 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:10.675509 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" event={"ID":"1fd003d5-b7f4-42bf-87ef-bfa1a035d906","Type":"ContainerStarted","Data":"4a3651f46ed1bbaa896d4b2902dc7bc05d0d33614d2d6ae99d501486d632f407"} Oct 31 14:54:10.940576 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:10.940491 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:11.896064 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:11.896032 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:11.944344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:11.944310 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:11.944682 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:11.944378 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:11.944682 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:11.944408 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:12.686455 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:12.686421 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" event={"ID":"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b","Type":"ContainerStarted","Data":"558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6"} Oct 31 14:54:12.858673 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:12.858622 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podStartSLOduration=2.291620683 podStartE2EDuration="5.858605403s" totalImagesPullingTime="3.56698472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:12.858129544 +0000 UTC m=+498.477680981" watchObservedRunningTime="2026-10-31 14:54:12.858605403 +0000 UTC m=+498.478156813" Oct 31 14:54:12.858855 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:12.858760 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" podStartSLOduration=4.433887702 podStartE2EDuration="6.858755584s" totalImagesPullingTime="2.424867882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:10.79399812 +0000 UTC m=+496.413549558" watchObservedRunningTime="2026-10-31 14:54:12.858755584 +0000 UTC m=+498.478307017" Oct 31 14:54:12.914645 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:12.914593 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:13.917094 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:13.917059 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:13.917731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:13.917701 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 31 14:54:14.914494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:14.913908 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:14.914494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:14.913971 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 31 14:54:15.234222 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.234183 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:54:15.482720 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.482676 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb"] Oct 31 14:54:15.486375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.486307 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.545704 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.545332 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb"] Oct 31 14:54:15.576048 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.576017 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:15.589216 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.589190 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.589375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.589223 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a447a957-8083-43b4-974e-0cc8f358cdba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.589375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.589247 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/a447a957-8083-43b4-974e-0cc8f358cdba-kube-api-access-f2vf4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.589375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.589269 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.589562 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.589414 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.589562 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.589501 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.593216 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.593186 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 14:54:15.690867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.690837 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.690867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.690871 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a447a957-8083-43b4-974e-0cc8f358cdba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.691128 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.690893 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/a447a957-8083-43b4-974e-0cc8f358cdba-kube-api-access-f2vf4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.691128 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.690918 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.691128 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.690942 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.691128 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.690972 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.691332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.691304 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.691382 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.691328 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.691533 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.691457 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.693444 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.693424 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a447a957-8083-43b4-974e-0cc8f358cdba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.699496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.693642 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a447a957-8083-43b4-974e-0cc8f358cdba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.753511 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.753416 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/a447a957-8083-43b4-974e-0cc8f358cdba-kube-api-access-f2vf4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb\" (UID: \"a447a957-8083-43b4-974e-0cc8f358cdba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.818872 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.818642 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:15.922507 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.919162 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:15.922507 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:15.919240 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 31 14:54:16.218925 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:16.209009 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb"] Oct 31 14:54:16.220351 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:54:16.220321 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda447a957_8083_43b4_974e_0cc8f358cdba.slice/crio-2461ff0051be1cf51f361068032db528f77a8a98729ade5c49a4c21317369e27 WatchSource:0}: Error finding container 2461ff0051be1cf51f361068032db528f77a8a98729ade5c49a4c21317369e27: Status 404 returned error can't find the container with id 2461ff0051be1cf51f361068032db528f77a8a98729ade5c49a4c21317369e27 Oct 31 14:54:16.738356 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:16.738323 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" event={"ID":"a447a957-8083-43b4-974e-0cc8f358cdba","Type":"ContainerStarted","Data":"2461ff0051be1cf51f361068032db528f77a8a98729ade5c49a4c21317369e27"} Oct 31 14:54:16.914498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:16.913843 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:16.914498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:16.913898 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 31 14:54:16.940282 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:16.940263 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:16.953275 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:16.953254 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:17.488882 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:54:17.488847 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:54:17.744050 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:17.743968 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" event={"ID":"a447a957-8083-43b4-974e-0cc8f358cdba","Type":"ContainerStarted","Data":"64c7c6144a491c8ad7c4745aa71821e3329dc495e7905c78eeca122a771bd986"} Oct 31 14:54:17.755318 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:17.755294 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" Oct 31 14:54:17.819351 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:17.819318 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:17.871992 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:17.871943 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podStartSLOduration=2.235286103 podStartE2EDuration="2.871927492s" totalImagesPullingTime="636.641389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:17.870260205 +0000 UTC m=+503.489811798" watchObservedRunningTime="2026-10-31 14:54:17.871927492 +0000 UTC m=+503.491478905" Oct 31 14:54:17.912149 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:17.912104 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:17.912358 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:17.912161 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 31 14:54:18.911934 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:18.911892 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:18.912367 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:18.911964 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Oct 31 14:54:20.912313 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:20.912273 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:54:20.912742 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:20.912338 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:54:21.954678 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:21.954633 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:54:21.955210 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:21.954707 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:54:22.911372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:22.911344 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:22.955697 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:22.955657 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:54:22.956095 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:22.955735 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:54:23.956899 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:23.956849 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:54:23.957429 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:23.956923 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:54:24.996443 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:24.996394 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:54:24.997616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:24.997585 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:54:24.997747 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:24.997643 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:54:25.602799 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:25.602756 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:25.604199 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:25.604174 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:54:25.818793 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:25.818765 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:25.831271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:25.831246 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:26.797484 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:26.797440 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 14:54:37.837904 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.837870 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx"] Oct 31 14:54:37.841381 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.841354 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.911821 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.911788 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.911821 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.911822 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.912041 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.911853 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.912041 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.912007 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.912127 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.912055 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzr5\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-kube-api-access-jzzr5\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.912127 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.912102 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.912231 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.912135 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.912231 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.912202 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:37.912336 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:37.912234 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013453 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013420 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013456 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzr5\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-kube-api-access-jzzr5\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013511 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013543 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013570 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013585 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013645 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013671 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.013998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.013700 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.014090 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.014021 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.014148 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.014130 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.014273 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.014170 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.014273 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.014221 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.014428 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.014410 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.016640 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.016613 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.016806 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.016791 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.038074 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.038048 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx"] Oct 31 14:54:38.071799 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.071771 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.071926 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.071853 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzr5\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-kube-api-access-jzzr5\") pod \"maas-default-gateway-openshift-default-67bf986f99-8l9gx\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.152985 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.152916 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:38.580355 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.580327 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx"] Oct 31 14:54:38.581672 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:54:38.581650 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8e00d7_d290_4ef8_86d3_14ebcbf34e4d.slice/crio-284de8653d1152b4ccce7100dcd5e45f7b7b70d3eabb7d93bb42d81ff0728eeb WatchSource:0}: Error finding container 284de8653d1152b4ccce7100dcd5e45f7b7b70d3eabb7d93bb42d81ff0728eeb: Status 404 returned error can't find the container with id 284de8653d1152b4ccce7100dcd5e45f7b7b70d3eabb7d93bb42d81ff0728eeb Oct 31 14:54:38.583879 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.583852 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:38.583968 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.583913 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:38.583968 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.583943 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:38.848101 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.848021 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" event={"ID":"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d","Type":"ContainerStarted","Data":"b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc"} Oct 31 14:54:38.848101 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.848078 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" event={"ID":"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d","Type":"ContainerStarted","Data":"284de8653d1152b4ccce7100dcd5e45f7b7b70d3eabb7d93bb42d81ff0728eeb"} Oct 31 14:54:38.967900 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:38.967838 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podStartSLOduration=1.967817473 podStartE2EDuration="1.967817473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:38.966966521 +0000 UTC m=+524.586517935" watchObservedRunningTime="2026-10-31 14:54:38.967817473 +0000 UTC m=+524.587368885" Oct 31 14:54:39.154253 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:39.154157 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:39.155820 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:39.155795 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:39.155953 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:39.155843 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:40.153640 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:40.153611 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:40.154038 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:40.153673 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:41.153702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:41.153670 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:41.154124 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:41.153731 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:42.153669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:42.153631 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:42.153855 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:42.153700 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:43.153679 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:43.153644 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:43.153861 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:43.153711 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:44.153376 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:44.153344 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:44.153575 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:44.153396 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:45.154299 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:45.154262 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:45.154712 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:45.154325 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:45.528052 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:45.528019 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:45.701112 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:45.701076 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:46.153279 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:46.153247 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:54:46.153420 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:46.153298 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 14:54:47.034379 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:47.034339 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:47.256705 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:47.256668 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:47.440740 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:47.440700 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:54:47.489803 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:54:47.489767 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:54:47.837601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:47.837572 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:47.838012 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:47.837989 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:47.838745 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:47.838699 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 14:54:48.376905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.376849 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-596f45b59f-4t67l"] Oct 31 14:54:48.377426 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.377159 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-596f45b59f-4t67l" podUID="2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" containerName="authorino" containerID="cri-o://3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6" gracePeriod=30 Oct 31 14:54:48.630367 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.630311 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:54:48.714319 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.714289 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-kube-api-access-zdsqd\" (UniqueName: \"kubernetes.io/projected/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-kube-api-access-zdsqd\") pod \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " Oct 31 14:54:48.714490 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.714433 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-tls-cert\") pod \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\" (UID: \"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e\") " Oct 31 14:54:48.716525 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.716500 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-kube-api-access-zdsqd" pod "2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" (UID: "2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e"). InnerVolumeSpecName "kube-api-access-zdsqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:54:48.724985 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.724962 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-tls-cert" pod "2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" (UID: "2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:54:48.815086 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.815056 4865 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-tls-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:54:48.815086 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.815084 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdsqd\" (UniqueName: \"kubernetes.io/projected/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e-kube-api-access-zdsqd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:54:48.896136 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.896052 4865 generic.go:356] "Generic (PLEG): container finished" podID="2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" containerID="3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6" exitCode=0 Oct 31 14:54:48.896136 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.896099 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-596f45b59f-4t67l" Oct 31 14:54:48.896332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.896132 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-596f45b59f-4t67l" event={"ID":"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e","Type":"ContainerDied","Data":"3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6"} Oct 31 14:54:48.896332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.896173 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-596f45b59f-4t67l" event={"ID":"2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e","Type":"ContainerDied","Data":"79eea8d246e34b6b33e18515a3ed10706c97e4dd39f4821fbdf3770ea2524ede"} Oct 31 14:54:48.896332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.896190 4865 scope.go:122] "RemoveContainer" containerID="3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6" Oct 31 14:54:48.905442 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.905424 4865 scope.go:122] "RemoveContainer" containerID="3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6" Oct 31 14:54:48.905799 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:54:48.905781 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6\": container with ID starting with 3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6 not found: ID does not exist" containerID="3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6" Oct 31 14:54:48.905869 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:48.905814 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6"} err="failed to get container status \"3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6\": rpc error: code = NotFound desc = could not find container \"3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6\": container with ID starting with 3b9f20d2c48ececf7c4a8067afffee4e55b238c5cd27f70de865b35dc5ae1ba6 not found: ID does not exist" Oct 31 14:54:49.066378 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:49.066345 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-596f45b59f-4t67l"] Oct 31 14:54:49.067804 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:49.067784 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-596f45b59f-4t67l"] Oct 31 14:54:50.496078 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:50.496039 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" path="/var/lib/kubelet/pods/2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e/volumes" Oct 31 14:54:52.935550 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:52.935516 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx"] Oct 31 14:54:52.935928 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:52.935915 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" containerName="authorino" Oct 31 14:54:52.935928 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:52.935925 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" containerName="authorino" Oct 31 14:54:52.935999 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:52.935988 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1a08f8-fb63-45b6-8fd0-c0b875f54e6e" containerName="authorino" Oct 31 14:54:52.940443 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:52.940426 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.039833 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.039803 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx"] Oct 31 14:54:53.052552 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052526 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052690 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052560 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052690 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052597 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d62163c-a533-4dfc-bb1e-533085ff1038-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052772 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052736 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052818 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052781 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052852 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4j7\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-kube-api-access-mz4j7\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052908 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052897 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052919 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.052980 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.052954 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.153990 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4j7\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-kube-api-access-mz4j7\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154193 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154078 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154193 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154117 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154193 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154145 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154366 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154191 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154366 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154233 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154501 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154407 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d62163c-a533-4dfc-bb1e-533085ff1038-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154501 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154444 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154501 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154492 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154674 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154546 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154796 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154770 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154914 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154794 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.154914 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.154859 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.155500 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.155447 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d62163c-a533-4dfc-bb1e-533085ff1038-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.156680 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.156660 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.157209 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.157191 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.223031 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.223002 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.227615 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.227586 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4j7\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-kube-api-access-mz4j7\") pod \"maas-default-gateway-openshift-default-67bf986f99-sjwtx\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.253002 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.252974 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:53.676374 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.676297 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx"] Oct 31 14:54:53.679369 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:54:53.679342 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d62163c_a533_4dfc_bb1e_533085ff1038.slice/crio-9de3fec3727e78901b9a9814a3ed90a96b348c6262d8a877fa2f6e2478b97c1b WatchSource:0}: Error finding container 9de3fec3727e78901b9a9814a3ed90a96b348c6262d8a877fa2f6e2478b97c1b: Status 404 returned error can't find the container with id 9de3fec3727e78901b9a9814a3ed90a96b348c6262d8a877fa2f6e2478b97c1b Oct 31 14:54:53.681335 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.681302 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:53.681422 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.681386 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:53.681422 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.681414 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:54:53.920974 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.920932 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" event={"ID":"2d62163c-a533-4dfc-bb1e-533085ff1038","Type":"ContainerStarted","Data":"e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1"} Oct 31 14:54:53.921132 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:53.921007 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" event={"ID":"2d62163c-a533-4dfc-bb1e-533085ff1038","Type":"ContainerStarted","Data":"9de3fec3727e78901b9a9814a3ed90a96b348c6262d8a877fa2f6e2478b97c1b"} Oct 31 14:54:54.041120 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:54.041064 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podStartSLOduration=2.04104897 podStartE2EDuration="2.04104897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:54.040133877 +0000 UTC m=+539.659685290" watchObservedRunningTime="2026-10-31 14:54:54.04104897 +0000 UTC m=+539.660600380" Oct 31 14:54:54.253667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:54.253633 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:54:54.255276 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:54.255241 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:54:54.255416 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:54.255291 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:54:55.253756 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:55.253719 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:54:55.254142 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:55.253777 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:54:56.253629 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:56.253595 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:54:56.253800 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:56.253645 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:54:57.253579 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:57.253542 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:54:57.253735 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:57.253597 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:54:58.253453 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:58.253418 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:54:58.253900 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:58.253497 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:54:59.253847 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:59.253808 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:54:59.254251 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:54:59.253866 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:55:00.253978 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:00.253942 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:55:00.254387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:00.254002 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:55:01.254311 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:01.254278 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sjwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:55:01.254840 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:01.254338 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 14:55:02.869244 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:02.869210 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:55:02.869719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:02.869519 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:55:02.870363 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:02.870345 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:55:09.762491 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:09.762423 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:17.488918 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:55:17.488869 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:55:17.657759 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:17.657615 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:17.821335 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:17.821263 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:17.971674 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:17.971613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:18.178191 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.178113 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:18.345600 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.345568 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx"] Oct 31 14:55:18.348940 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.348917 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395509 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-envoy\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395687 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-credential-socket\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395731 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-certs\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395751 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istiod-ca-cert\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395786 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-socket\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395808 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-data\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395828 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-podinfo\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395856 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4fz7\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-kube-api-access-l4fz7\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.401527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.395960 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-token\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.428592 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.428524 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-08b12cc9-openshift-default-dockercfg-dk5hw\"" Oct 31 14:55:18.436310 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.436277 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:18.455161 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.455127 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx"] Oct 31 14:55:18.497015 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.496986 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-token\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497030 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-envoy\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497078 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-credential-socket\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497100 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-certs\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497128 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istiod-ca-cert\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497270 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-socket\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497311 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-data\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497336 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-podinfo\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497360 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4fz7\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-kube-api-access-l4fz7\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497973 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497576 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-certs\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497973 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497787 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-credential-socket\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.497973 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.497862 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-socket\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.498195 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.498037 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-data\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.498195 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.498094 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istiod-ca-cert\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.499806 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.499778 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-envoy\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.500585 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.500560 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-podinfo\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.558430 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.558395 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-token\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.558657 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.558540 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4fz7\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-kube-api-access-l4fz7\") pod \"e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.583816 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.583781 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:18.661442 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.661414 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:18.767173 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:18.767138 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:19.035766 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.035680 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:19.066849 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.066810 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp"] Oct 31 14:55:19.071331 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.071311 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.100533 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.100511 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx"] Oct 31 14:55:19.102996 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:19.102971 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6f03e0_85c1_4ba2_8c64_98dc0f56130b.slice/crio-7a3a715cff8c08aa7eed8271014f2154659ca89a25672099972409cf7f193ce3 WatchSource:0}: Error finding container 7a3a715cff8c08aa7eed8271014f2154659ca89a25672099972409cf7f193ce3: Status 404 returned error can't find the container with id 7a3a715cff8c08aa7eed8271014f2154659ca89a25672099972409cf7f193ce3 Oct 31 14:55:19.103457 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103166 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-envoy\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.103457 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103378 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-socket\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.103670 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103501 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-credential-socket\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.103670 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103556 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858acbb3-a143-4c16-a03e-cd46cfeb4137-istiod-ca-cert\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.103902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103697 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sx86\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-kube-api-access-8sx86\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.103902 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103735 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-podinfo\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.104010 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103764 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-data\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.104010 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.103995 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-token\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.104116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.104034 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-certs\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.108731 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.108701 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:19.108845 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.108759 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:19.108845 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.108798 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:19.128230 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.128200 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b1c2330c-gw-openshift-default-dockercfg-st9jg\"" Oct 31 14:55:19.130257 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.130232 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp"] Oct 31 14:55:19.192253 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.191806 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:19.205401 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205372 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858acbb3-a143-4c16-a03e-cd46cfeb4137-istiod-ca-cert\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.205616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205426 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sx86\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-kube-api-access-8sx86\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.205616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205459 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-podinfo\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.205616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205515 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-data\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.205616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205530 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-token\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.205616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205556 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-certs\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.205616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205596 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-envoy\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.211702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205627 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-socket\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.211702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.205657 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-credential-socket\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.211702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.206060 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-credential-socket\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.211702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.206736 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858acbb3-a143-4c16-a03e-cd46cfeb4137-istiod-ca-cert\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.211702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.207607 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-certs\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.211702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.207866 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-data\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.211702 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.211588 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-envoy\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.212055 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.211828 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-socket\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.214371 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.214313 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-podinfo\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.265293 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.265256 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sx86\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-kube-api-access-8sx86\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.266195 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.266165 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-token\") pod \"e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.385720 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.385627 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:19.389544 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.389501 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:19.812223 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.812204 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp"] Oct 31 14:55:19.813709 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:19.813683 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858acbb3_a143_4c16_a03e_cd46cfeb4137.slice/crio-d44cd9334e5ac1e87fe8a7e0585b94ad88b7817b2e97b14ae214d6d52f161e0d WatchSource:0}: Error finding container d44cd9334e5ac1e87fe8a7e0585b94ad88b7817b2e97b14ae214d6d52f161e0d: Status 404 returned error can't find the container with id d44cd9334e5ac1e87fe8a7e0585b94ad88b7817b2e97b14ae214d6d52f161e0d Oct 31 14:55:19.816774 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.816743 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:19.816856 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.816809 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:19.816856 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:19.816842 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:20.041056 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.041020 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" event={"ID":"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b","Type":"ContainerStarted","Data":"280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385"} Oct 31 14:55:20.041257 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.041083 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" event={"ID":"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b","Type":"ContainerStarted","Data":"7a3a715cff8c08aa7eed8271014f2154659ca89a25672099972409cf7f193ce3"} Oct 31 14:55:20.043172 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.043133 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" event={"ID":"858acbb3-a143-4c16-a03e-cd46cfeb4137","Type":"ContainerStarted","Data":"19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9"} Oct 31 14:55:20.043320 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.043193 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" event={"ID":"858acbb3-a143-4c16-a03e-cd46cfeb4137","Type":"ContainerStarted","Data":"d44cd9334e5ac1e87fe8a7e0585b94ad88b7817b2e97b14ae214d6d52f161e0d"} Oct 31 14:55:20.160202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.160098 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podStartSLOduration=2.160082421 podStartE2EDuration="2.160082421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:20.159391453 +0000 UTC m=+565.778942898" watchObservedRunningTime="2026-10-31 14:55:20.160082421 +0000 UTC m=+565.779633823" Oct 31 14:55:20.281496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.281243 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podStartSLOduration=1.281221609 podStartE2EDuration="1.281221609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:20.280068255 +0000 UTC m=+565.899619697" watchObservedRunningTime="2026-10-31 14:55:20.281221609 +0000 UTC m=+565.900773020" Oct 31 14:55:20.386606 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.386572 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:20.407933 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.407898 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:20.477978 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.477946 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:20.662356 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.662326 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:20.667188 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.667165 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:20.975913 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:20.975884 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:21.064117 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:21.064022 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:21.064117 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:21.064084 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:21.065263 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:21.065226 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:21.065750 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:21.065730 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:55:21.208319 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:21.208287 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:22.195626 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:22.195597 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:22.358229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:22.358191 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:22.493674 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:22.493644 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:23.120046 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:23.120016 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:23.312512 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:23.312462 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:23.695995 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:23.695958 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:23.890522 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:23.890460 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:24.746669 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:24.746636 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:24.925351 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:24.925325 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:25.067932 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:25.067841 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:25.669517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:25.669461 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:25.902563 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:25.902529 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:26.119766 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.119734 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:26.289950 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.289921 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:26.538762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.538728 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd"] Oct 31 14:55:26.542301 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.542284 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.581083 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.581049 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85747f29-f7af-42a8-9ef4-7caf1483798c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.581260 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.581116 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpcb\" (UniqueName: \"kubernetes.io/projected/85747f29-f7af-42a8-9ef4-7caf1483798c-kube-api-access-8jpcb\") pod \"payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.599263 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.599229 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd"] Oct 31 14:55:26.648021 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.647997 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b1c2330c\"" Oct 31 14:55:26.685109 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.685043 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85747f29-f7af-42a8-9ef4-7caf1483798c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.685338 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.685153 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpcb\" (UniqueName: \"kubernetes.io/projected/85747f29-f7af-42a8-9ef4-7caf1483798c-kube-api-access-8jpcb\") pod \"payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.686368 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.686336 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85747f29-f7af-42a8-9ef4-7caf1483798c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.718281 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.718245 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b1c2330c-dockercfg-l98px\"" Oct 31 14:55:26.719117 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.719088 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn"] Oct 31 14:55:26.725559 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.724926 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:26.744768 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.744737 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpcb\" (UniqueName: \"kubernetes.io/projected/85747f29-f7af-42a8-9ef4-7caf1483798c-kube-api-access-8jpcb\") pod \"payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.752556 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.752426 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn"] Oct 31 14:55:26.785853 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.785825 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str9w\" (UniqueName: \"kubernetes.io/projected/9b1c04e3-142a-406d-8141-7e2864764cdb-kube-api-access-str9w\") pod \"payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:26.786016 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.785977 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c04e3-142a-406d-8141-7e2864764cdb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:26.854832 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.854758 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:26.886545 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.886515 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str9w\" (UniqueName: \"kubernetes.io/projected/9b1c04e3-142a-406d-8141-7e2864764cdb-kube-api-access-str9w\") pod \"payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:26.886751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.886623 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c04e3-142a-406d-8141-7e2864764cdb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:26.887328 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.887306 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c04e3-142a-406d-8141-7e2864764cdb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:26.945463 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:26.945404 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-str9w\" (UniqueName: \"kubernetes.io/projected/9b1c04e3-142a-406d-8141-7e2864764cdb-kube-api-access-str9w\") pod \"payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:27.043456 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:27.043416 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:27.226121 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:27.226093 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd"] Oct 31 14:55:27.227139 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:27.227102 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85747f29_f7af_42a8_9ef4_7caf1483798c.slice/crio-d09d21e6349532c04e55dd8a997da423d819a3b0f02eb1a581097c2b3fe28a45 WatchSource:0}: Error finding container d09d21e6349532c04e55dd8a997da423d819a3b0f02eb1a581097c2b3fe28a45: Status 404 returned error can't find the container with id d09d21e6349532c04e55dd8a997da423d819a3b0f02eb1a581097c2b3fe28a45 Oct 31 14:55:27.413196 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:27.413172 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn"] Oct 31 14:55:27.414396 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:27.414373 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1c04e3_142a_406d_8141_7e2864764cdb.slice/crio-568855ba326da80e4c8a701e777284d7b4f6502d3c6d5d72fe5e99671e932658 WatchSource:0}: Error finding container 568855ba326da80e4c8a701e777284d7b4f6502d3c6d5d72fe5e99671e932658: Status 404 returned error can't find the container with id 568855ba326da80e4c8a701e777284d7b4f6502d3c6d5d72fe5e99671e932658 Oct 31 14:55:28.093015 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:28.092971 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" event={"ID":"9b1c04e3-142a-406d-8141-7e2864764cdb","Type":"ContainerStarted","Data":"568855ba326da80e4c8a701e777284d7b4f6502d3c6d5d72fe5e99671e932658"} Oct 31 14:55:28.094231 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:28.094200 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" event={"ID":"85747f29-f7af-42a8-9ef4-7caf1483798c","Type":"ContainerStarted","Data":"d09d21e6349532c04e55dd8a997da423d819a3b0f02eb1a581097c2b3fe28a45"} Oct 31 14:55:28.760410 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:28.760365 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:28.888009 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:28.887976 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:29.030174 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:29.030069 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:29.158111 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:29.158082 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:30.872248 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:30.872217 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:31.042486 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.042440 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:31.125956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.125859 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" event={"ID":"9b1c04e3-142a-406d-8141-7e2864764cdb","Type":"ContainerStarted","Data":"54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d"} Oct 31 14:55:31.125956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.125925 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:31.127377 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.127353 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" event={"ID":"85747f29-f7af-42a8-9ef4-7caf1483798c","Type":"ContainerStarted","Data":"e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a"} Oct 31 14:55:31.127522 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.127463 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:31.242331 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.242283 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" podStartSLOduration=2.424176199 podStartE2EDuration="5.242270413s" totalImagesPullingTime="2.818094214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:31.241804074 +0000 UTC m=+576.861355490" watchObservedRunningTime="2026-10-31 14:55:31.242270413 +0000 UTC m=+576.861821860" Oct 31 14:55:31.294373 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.294333 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh"] Oct 31 14:55:31.297994 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.297979 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.352136 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.351173 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-08b12cc9\"" Oct 31 14:55:31.365091 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.352961 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-08b12cc9-dockercfg-2bj6b\"" Oct 31 14:55:31.365091 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.354528 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh"] Oct 31 14:55:31.371628 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.368037 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" podStartSLOduration=2.455062484 podStartE2EDuration="5.368017783s" totalImagesPullingTime="2.912955299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:31.365881783 +0000 UTC m=+576.985433195" watchObservedRunningTime="2026-10-31 14:55:31.368017783 +0000 UTC m=+576.987569195" Oct 31 14:55:31.434832 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.434746 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr"] Oct 31 14:55:31.437217 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.437183 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dss2n\" (UniqueName: \"kubernetes.io/projected/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-kube-api-access-dss2n\") pod \"payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.437391 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.437331 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.438843 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.438820 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.494703 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.494672 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr"] Oct 31 14:55:31.538332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.538301 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn2nw\" (UniqueName: \"kubernetes.io/projected/46ff68b4-004d-4e45-958d-0d3fc91612ac-kube-api-access-cn2nw\") pod \"payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.538512 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.538342 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.538569 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.538514 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dss2n\" (UniqueName: \"kubernetes.io/projected/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-kube-api-access-dss2n\") pod \"payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.538569 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.538544 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ff68b4-004d-4e45-958d-0d3fc91612ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.539144 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.539126 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.600521 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.600494 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dss2n\" (UniqueName: \"kubernetes.io/projected/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-kube-api-access-dss2n\") pod \"payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.610158 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.610136 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:31.639942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.639915 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ff68b4-004d-4e45-958d-0d3fc91612ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.640046 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.639967 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2nw\" (UniqueName: \"kubernetes.io/projected/46ff68b4-004d-4e45-958d-0d3fc91612ac-kube-api-access-cn2nw\") pod \"payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.640783 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.640762 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ff68b4-004d-4e45-958d-0d3fc91612ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.698027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.697965 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn2nw\" (UniqueName: \"kubernetes.io/projected/46ff68b4-004d-4e45-958d-0d3fc91612ac-kube-api-access-cn2nw\") pod \"payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.751525 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.751455 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:31.982395 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:31.982367 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh"] Oct 31 14:55:31.983255 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:31.983230 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda164f69f_11b2_4d3d_b7c8_a57fa1c9a5c4.slice/crio-493af71b8acea25bc99b499820d6f7dc17ae9b10c82aaafe97c0bf420711a5f8 WatchSource:0}: Error finding container 493af71b8acea25bc99b499820d6f7dc17ae9b10c82aaafe97c0bf420711a5f8: Status 404 returned error can't find the container with id 493af71b8acea25bc99b499820d6f7dc17ae9b10c82aaafe97c0bf420711a5f8 Oct 31 14:55:32.095040 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.095010 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:32.125299 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.125277 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr"] Oct 31 14:55:32.127011 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:32.126979 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ff68b4_004d_4e45_958d_0d3fc91612ac.slice/crio-af8250052a9e6cb5cad1fff76253bb44c799825da43b9ab3a4c81b938cf11b73 WatchSource:0}: Error finding container af8250052a9e6cb5cad1fff76253bb44c799825da43b9ab3a4c81b938cf11b73: Status 404 returned error can't find the container with id af8250052a9e6cb5cad1fff76253bb44c799825da43b9ab3a4c81b938cf11b73 Oct 31 14:55:32.133953 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.133926 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" event={"ID":"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4","Type":"ContainerStarted","Data":"f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223"} Oct 31 14:55:32.134098 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.134082 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:32.134162 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.134121 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" event={"ID":"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4","Type":"ContainerStarted","Data":"493af71b8acea25bc99b499820d6f7dc17ae9b10c82aaafe97c0bf420711a5f8"} Oct 31 14:55:32.135250 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.135228 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" event={"ID":"46ff68b4-004d-4e45-958d-0d3fc91612ac","Type":"ContainerStarted","Data":"af8250052a9e6cb5cad1fff76253bb44c799825da43b9ab3a4c81b938cf11b73"} Oct 31 14:55:32.251483 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.251386 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" podStartSLOduration=1.251370259 podStartE2EDuration="1.251370259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:32.250328877 +0000 UTC m=+577.869880301" watchObservedRunningTime="2026-10-31 14:55:32.251370259 +0000 UTC m=+577.870921673" Oct 31 14:55:32.319191 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.319163 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:32.450285 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:32.450224 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:33.154011 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.153976 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" event={"ID":"46ff68b4-004d-4e45-958d-0d3fc91612ac","Type":"ContainerStarted","Data":"85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862"} Oct 31 14:55:33.154424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.154118 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:33.261875 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.261845 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:55:33.289230 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.289180 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" podStartSLOduration=2.289166319 podStartE2EDuration="2.289166319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:33.287635852 +0000 UTC m=+578.907187260" watchObservedRunningTime="2026-10-31 14:55:33.289166319 +0000 UTC m=+578.908717730" Oct 31 14:55:33.336653 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.336621 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:33.495267 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.495236 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:33.860657 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.860580 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks"] Oct 31 14:55:33.864336 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.864317 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:33.919905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.919876 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-33bc513e\"" Oct 31 14:55:33.922435 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.922409 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks"] Oct 31 14:55:33.963314 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.963278 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-plugins-config-volume\") pod \"payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:33.963503 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.963412 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9ll\" (UniqueName: \"kubernetes.io/projected/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-kube-api-access-gq9ll\") pod \"payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:33.989740 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:33.989709 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-33bc513e-dockercfg-7tgs2\"" Oct 31 14:55:34.005214 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.005183 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:34.067494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.067077 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-plugins-config-volume\") pod \"payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:34.067494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.067184 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9ll\" (UniqueName: \"kubernetes.io/projected/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-kube-api-access-gq9ll\") pod \"payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:34.069400 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.068531 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-plugins-config-volume\") pod \"payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:34.125482 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.125404 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9ll\" (UniqueName: \"kubernetes.io/projected/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-kube-api-access-gq9ll\") pod \"payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:34.151121 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.151065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:34.178422 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.178395 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:34.284917 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.284890 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:34.563930 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:34.563909 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks"] Oct 31 14:55:34.565928 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:34.565904 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0febb63a_d8c7_48f7_8659_bf5da5a7ad81.slice/crio-ba149e0d93cd8a19441c18d1ce0c5a46b65c42c1e38deb8667bd2a5f2c1c9ff0 WatchSource:0}: Error finding container ba149e0d93cd8a19441c18d1ce0c5a46b65c42c1e38deb8667bd2a5f2c1c9ff0: Status 404 returned error can't find the container with id ba149e0d93cd8a19441c18d1ce0c5a46b65c42c1e38deb8667bd2a5f2c1c9ff0 Oct 31 14:55:35.164515 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:35.164482 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" event={"ID":"0febb63a-d8c7-48f7-8659-bf5da5a7ad81","Type":"ContainerStarted","Data":"ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738"} Oct 31 14:55:35.164719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:35.164626 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:35.164719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:35.164679 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" event={"ID":"0febb63a-d8c7-48f7-8659-bf5da5a7ad81","Type":"ContainerStarted","Data":"ba149e0d93cd8a19441c18d1ce0c5a46b65c42c1e38deb8667bd2a5f2c1c9ff0"} Oct 31 14:55:35.282295 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:35.282209 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" podStartSLOduration=2.282189388 podStartE2EDuration="2.282189388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:35.280071176 +0000 UTC m=+580.899622588" watchObservedRunningTime="2026-10-31 14:55:35.282189388 +0000 UTC m=+580.901740848" Oct 31 14:55:35.835602 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:35.835571 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:36.009309 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.009280 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:36.414752 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.414721 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk"] Oct 31 14:55:36.418305 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.418289 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:36.488794 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.488769 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2dba94-8927-487b-9bcf-436713422469-plugins-config-volume\") pod \"payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:36.488947 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.488895 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2cf\" (UniqueName: \"kubernetes.io/projected/ac2dba94-8927-487b-9bcf-436713422469-kube-api-access-cj2cf\") pod \"payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:36.524054 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.524020 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b671479d\"" Oct 31 14:55:36.524205 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.524067 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b671479d-dockercfg-t7bsk\"" Oct 31 14:55:36.528504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.526531 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk"] Oct 31 14:55:36.589988 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.589956 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2cf\" (UniqueName: \"kubernetes.io/projected/ac2dba94-8927-487b-9bcf-436713422469-kube-api-access-cj2cf\") pod \"payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:36.590159 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.590041 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2dba94-8927-487b-9bcf-436713422469-plugins-config-volume\") pod \"payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:36.590807 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.590788 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2dba94-8927-487b-9bcf-436713422469-plugins-config-volume\") pod \"payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:36.647822 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.647794 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2cf\" (UniqueName: \"kubernetes.io/projected/ac2dba94-8927-487b-9bcf-436713422469-kube-api-access-cj2cf\") pod \"payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:36.731275 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:36.731248 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:37.121217 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:37.121191 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk"] Oct 31 14:55:37.122837 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:37.122811 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2dba94_8927_487b_9bcf_436713422469.slice/crio-93685e4b5c4565afb12333ed60a52e79f07b589aae669e6e5ed0dc0cacbaf3bf WatchSource:0}: Error finding container 93685e4b5c4565afb12333ed60a52e79f07b589aae669e6e5ed0dc0cacbaf3bf: Status 404 returned error can't find the container with id 93685e4b5c4565afb12333ed60a52e79f07b589aae669e6e5ed0dc0cacbaf3bf Oct 31 14:55:37.176695 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:37.176664 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" event={"ID":"ac2dba94-8927-487b-9bcf-436713422469","Type":"ContainerStarted","Data":"93685e4b5c4565afb12333ed60a52e79f07b589aae669e6e5ed0dc0cacbaf3bf"} Oct 31 14:55:38.185762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:38.185724 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" event={"ID":"ac2dba94-8927-487b-9bcf-436713422469","Type":"ContainerStarted","Data":"ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3"} Oct 31 14:55:38.186179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:38.185852 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:38.307958 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:38.307904 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" podStartSLOduration=2.307889896 podStartE2EDuration="2.307889896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:38.306424351 +0000 UTC m=+583.925975763" watchObservedRunningTime="2026-10-31 14:55:38.307889896 +0000 UTC m=+583.927441306" Oct 31 14:55:39.187951 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:39.187917 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:39.343897 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:39.343871 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:39.503787 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:39.503750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:42.138260 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:42.138232 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:55:42.138747 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:42.138300 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:55:43.157326 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:43.157297 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:55:43.793062 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:43.793029 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:44.161141 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:44.161066 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:55:45.265421 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.265386 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:45.439238 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.439205 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:45.838082 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.838049 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc"] Oct 31 14:55:45.841830 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.841809 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:45.898621 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.898592 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc"] Oct 31 14:55:45.947028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.947002 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ee109dd4\"" Oct 31 14:55:45.947209 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.947191 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ee109dd4-dockercfg-p9nr9\"" Oct 31 14:55:45.996233 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.996201 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53cb03d-60fb-489f-aa69-def6ab054ae7-plugins-config-volume\") pod \"payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:45.996371 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:45.996251 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2ds\" (UniqueName: \"kubernetes.io/projected/c53cb03d-60fb-489f-aa69-def6ab054ae7-kube-api-access-7w2ds\") pod \"payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:46.097399 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:46.097327 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53cb03d-60fb-489f-aa69-def6ab054ae7-plugins-config-volume\") pod \"payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:46.097399 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:46.097384 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2ds\" (UniqueName: \"kubernetes.io/projected/c53cb03d-60fb-489f-aa69-def6ab054ae7-kube-api-access-7w2ds\") pod \"payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:46.098079 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:46.098060 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53cb03d-60fb-489f-aa69-def6ab054ae7-plugins-config-volume\") pod \"payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:46.155927 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:46.155900 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2ds\" (UniqueName: \"kubernetes.io/projected/c53cb03d-60fb-489f-aa69-def6ab054ae7-kube-api-access-7w2ds\") pod \"payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:46.173704 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:46.173675 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:55:46.454949 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:46.454858 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:46.830174 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:46.830145 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc"] Oct 31 14:55:46.831124 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:55:46.831098 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53cb03d_60fb_489f_aa69_def6ab054ae7.slice/crio-96407c3407e2257988bdd75d868bcc82ecc528cf4cf301391c61d1c2c775e4ab WatchSource:0}: Error finding container 96407c3407e2257988bdd75d868bcc82ecc528cf4cf301391c61d1c2c775e4ab: Status 404 returned error can't find the container with id 96407c3407e2257988bdd75d868bcc82ecc528cf4cf301391c61d1c2c775e4ab Oct 31 14:55:47.237073 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:47.237035 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" event={"ID":"c53cb03d-60fb-489f-aa69-def6ab054ae7","Type":"ContainerStarted","Data":"3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607"} Oct 31 14:55:47.237263 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:47.237098 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" event={"ID":"c53cb03d-60fb-489f-aa69-def6ab054ae7","Type":"ContainerStarted","Data":"96407c3407e2257988bdd75d868bcc82ecc528cf4cf301391c61d1c2c775e4ab"} Oct 31 14:55:47.237263 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:47.237236 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:47.357856 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:47.357807 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" podStartSLOduration=2.357794961 podStartE2EDuration="2.357794961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:47.354672469 +0000 UTC m=+592.974223880" watchObservedRunningTime="2026-10-31 14:55:47.357794961 +0000 UTC m=+592.977346371" Oct 31 14:55:47.490886 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:55:47.490791 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:55:47.628275 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:47.628236 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:47.828672 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:47.828596 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:55:48.645915 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:48.645877 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:48.776773 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:48.776738 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:48.919940 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:48.919847 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:49.041715 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:49.041670 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:49.216692 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:49.216660 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:55:53.443952 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:53.443912 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:53.617502 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:53.616830 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:53.738284 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:53.738258 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:54.538524 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:54.538496 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 14:55:54.547570 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:54.547542 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 14:55:54.548387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:54.548356 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:55:54.556289 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:54.556270 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 14:55:56.658078 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:56.658020 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:58.246245 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.246214 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:55:58.316494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.293622 4865 generic.go:356] "Generic (PLEG): container finished" podID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerID="19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9" exitCode=0 Oct 31 14:55:58.316494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.293766 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" event={"ID":"858acbb3-a143-4c16-a03e-cd46cfeb4137","Type":"ContainerDied","Data":"19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9"} Oct 31 14:55:58.316494 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.294140 4865 scope.go:122] "RemoveContainer" containerID="19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9" Oct 31 14:55:58.319300 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.319258 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:55:58.319692 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.319666 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:58.319762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.319731 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:58.319799 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:58.319762 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:55:59.326717 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:59.325706 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" event={"ID":"858acbb3-a143-4c16-a03e-cd46cfeb4137","Type":"ContainerStarted","Data":"053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142"} Oct 31 14:55:59.386212 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:59.386055 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:55:59.386424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:59.386301 4865 patch_prober.go:29] interesting pod/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 14:55:59.386424 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:55:59.386366 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 14:56:00.386792 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:00.386759 4865 patch_prober.go:29] interesting pod/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 14:56:00.387221 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:00.386809 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 14:56:01.386663 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:01.386630 4865 patch_prober.go:29] interesting pod/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 14:56:01.386851 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:01.386681 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 14:56:02.386580 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:02.386542 4865 patch_prober.go:29] interesting pod/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 14:56:02.386780 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:02.386596 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 14:56:02.721406 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:02.721361 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:56:03.147543 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:03.147434 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:03.386537 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:03.386502 4865 patch_prober.go:29] interesting pod/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 14:56:03.386718 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:03.386559 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 14:56:04.386269 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:04.386227 4865 patch_prober.go:29] interesting pod/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Oct 31 14:56:04.386843 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:04.386281 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Oct 31 14:56:05.390721 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:05.390691 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:56:05.391208 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:05.391042 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:56:05.391648 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:05.391627 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:56:06.065458 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.065424 4865 patch_prober.go:29] interesting pod/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:06.065686 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.065516 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 14:56:06.375980 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.375892 4865 generic.go:356] "Generic (PLEG): container finished" podID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerID="280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385" exitCode=0 Oct 31 14:56:06.375980 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.375967 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" event={"ID":"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b","Type":"ContainerDied","Data":"280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385"} Oct 31 14:56:06.376346 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.376326 4865 scope.go:122] "RemoveContainer" containerID="280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385" Oct 31 14:56:06.377666 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.377641 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:56:06.377732 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.377708 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:56:06.377769 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:06.377739 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:56:07.385563 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:07.385510 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" event={"ID":"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b","Type":"ContainerStarted","Data":"18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e"} Oct 31 14:56:07.662325 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:07.662244 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:56:07.662501 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:07.662419 4865 patch_prober.go:29] interesting pod/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:07.662558 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:07.662501 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 14:56:08.091813 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:08.091778 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:08.662758 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:08.662725 4865 patch_prober.go:29] interesting pod/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:08.663199 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:08.662774 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 14:56:09.045305 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.045273 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766"] Oct 31 14:56:09.048946 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.048928 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.111137 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.111101 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766"] Oct 31 14:56:09.140797 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.140765 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.140966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.140828 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.141020 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.140958 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqml\" (UniqueName: \"kubernetes.io/projected/9c2b46f3-74f4-4e65-88c5-e97dca098742-kube-api-access-2zqml\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.158924 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.158902 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:56:09.158924 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.158903 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:56:09.159110 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.158904 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:56:09.159110 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.158971 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 14:56:09.241793 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.241762 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqml\" (UniqueName: \"kubernetes.io/projected/9c2b46f3-74f4-4e65-88c5-e97dca098742-kube-api-access-2zqml\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.241999 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.241871 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.241999 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.241973 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.242599 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.242581 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.244403 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.244383 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.318548 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.316859 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqml\" (UniqueName: \"kubernetes.io/projected/9c2b46f3-74f4-4e65-88c5-e97dca098742-kube-api-access-2zqml\") pod \"maas-api-revoke-keys-e2e-ait-33bc513e-jz766\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.361921 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.361896 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:09.662297 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.662223 4865 patch_prober.go:29] interesting pod/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:09.662297 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.662271 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 14:56:09.757565 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:09.757538 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766"] Oct 31 14:56:09.758504 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:09.758459 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2b46f3_74f4_4e65_88c5_e97dca098742.slice/crio-6236ba145adeb143a76aeb2a3aa5ba60ec39d40669e84b00371f4a8c992a44f0 WatchSource:0}: Error finding container 6236ba145adeb143a76aeb2a3aa5ba60ec39d40669e84b00371f4a8c992a44f0: Status 404 returned error can't find the container with id 6236ba145adeb143a76aeb2a3aa5ba60ec39d40669e84b00371f4a8c992a44f0 Oct 31 14:56:10.424330 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:10.424292 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" event={"ID":"9c2b46f3-74f4-4e65-88c5-e97dca098742","Type":"ContainerStarted","Data":"6236ba145adeb143a76aeb2a3aa5ba60ec39d40669e84b00371f4a8c992a44f0"} Oct 31 14:56:10.662591 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:10.662556 4865 patch_prober.go:29] interesting pod/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:10.662764 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:10.662614 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 14:56:11.662196 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:11.662167 4865 patch_prober.go:29] interesting pod/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:11.662626 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:11.662236 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 31 14:56:12.666873 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:12.666843 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:56:12.667391 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:12.667240 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:56:12.668181 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:12.668159 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:56:12.985966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:12.985936 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:14.449757 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:14.449724 4865 generic.go:356] "Generic (PLEG): container finished" podID="9c2b46f3-74f4-4e65-88c5-e97dca098742" containerID="a5972ae7bdbf21e638fc65d9d5082a525356f2b5e7ba64fbea7e84985bb8cba1" exitCode=0 Oct 31 14:56:14.450172 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:14.449807 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" event={"ID":"9c2b46f3-74f4-4e65-88c5-e97dca098742","Type":"ContainerDied","Data":"a5972ae7bdbf21e638fc65d9d5082a525356f2b5e7ba64fbea7e84985bb8cba1"} Oct 31 14:56:15.591143 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.591116 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:15.608219 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.608196 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-xks-ca\") pod \"9c2b46f3-74f4-4e65-88c5-e97dca098742\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " Oct 31 14:56:15.608315 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.608280 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-openshift-ca\") pod \"9c2b46f3-74f4-4e65-88c5-e97dca098742\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " Oct 31 14:56:15.608391 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.608367 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2b46f3-74f4-4e65-88c5-e97dca098742-kube-api-access-2zqml\" (UniqueName: \"kubernetes.io/projected/9c2b46f3-74f4-4e65-88c5-e97dca098742-kube-api-access-2zqml\") pod \"9c2b46f3-74f4-4e65-88c5-e97dca098742\" (UID: \"9c2b46f3-74f4-4e65-88c5-e97dca098742\") " Oct 31 14:56:15.608716 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.608690 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-openshift-ca" pod "9c2b46f3-74f4-4e65-88c5-e97dca098742" (UID: "9c2b46f3-74f4-4e65-88c5-e97dca098742"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:15.610598 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.610577 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-xks-ca" pod "9c2b46f3-74f4-4e65-88c5-e97dca098742" (UID: "9c2b46f3-74f4-4e65-88c5-e97dca098742"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:56:15.610598 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.610585 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2b46f3-74f4-4e65-88c5-e97dca098742-kube-api-access-2zqml" pod "9c2b46f3-74f4-4e65-88c5-e97dca098742" (UID: "9c2b46f3-74f4-4e65-88c5-e97dca098742"). InnerVolumeSpecName "kube-api-access-2zqml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:15.709625 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.709559 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zqml\" (UniqueName: \"kubernetes.io/projected/9c2b46f3-74f4-4e65-88c5-e97dca098742-kube-api-access-2zqml\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:15.709625 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.709596 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:15.709625 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:15.709611 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b46f3-74f4-4e65-88c5-e97dca098742-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:16.461334 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:16.461306 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" Oct 31 14:56:16.461504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:16.461347 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766" event={"ID":"9c2b46f3-74f4-4e65-88c5-e97dca098742","Type":"ContainerDied","Data":"6236ba145adeb143a76aeb2a3aa5ba60ec39d40669e84b00371f4a8c992a44f0"} Oct 31 14:56:16.461504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:16.461420 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6236ba145adeb143a76aeb2a3aa5ba60ec39d40669e84b00371f4a8c992a44f0" Oct 31 14:56:17.489291 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:17.489255 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:56:18.149293 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.149232 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:18.514034 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.514007 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl"] Oct 31 14:56:18.514520 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.514445 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9c2b46f3-74f4-4e65-88c5-e97dca098742" containerName="revoke-keys" Oct 31 14:56:18.514520 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.514456 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2b46f3-74f4-4e65-88c5-e97dca098742" containerName="revoke-keys" Oct 31 14:56:18.514601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.514556 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2b46f3-74f4-4e65-88c5-e97dca098742" containerName="revoke-keys" Oct 31 14:56:18.517514 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.517497 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.573174 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.573145 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl"] Oct 31 14:56:18.621526 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.621504 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:56:18.621526 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.621505 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:56:18.621709 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.621505 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:56:18.622047 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.622032 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 14:56:18.636970 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.636949 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqb6\" (UniqueName: \"kubernetes.io/projected/3c0af175-3134-46fd-9446-66ef0f8b9950-kube-api-access-rhqb6\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.637097 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.636985 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.637144 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.637124 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.737831 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.737792 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqb6\" (UniqueName: \"kubernetes.io/projected/3c0af175-3134-46fd-9446-66ef0f8b9950-kube-api-access-rhqb6\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.737831 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.737839 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.738048 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.737905 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.738526 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.738505 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.740455 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.740429 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.796039 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.795961 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqb6\" (UniqueName: \"kubernetes.io/projected/3c0af175-3134-46fd-9446-66ef0f8b9950-kube-api-access-rhqb6\") pod \"maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:18.829995 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:18.829965 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:19.201267 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:19.201243 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl"] Oct 31 14:56:19.202413 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:19.202390 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0af175_3134_46fd_9446_66ef0f8b9950.slice/crio-6dad3d0dfd54d745ca39003174cc3cda55ac9a6dfdc1cec01e7ce6b0fbab7700 WatchSource:0}: Error finding container 6dad3d0dfd54d745ca39003174cc3cda55ac9a6dfdc1cec01e7ce6b0fbab7700: Status 404 returned error can't find the container with id 6dad3d0dfd54d745ca39003174cc3cda55ac9a6dfdc1cec01e7ce6b0fbab7700 Oct 31 14:56:19.477733 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:19.477704 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" event={"ID":"3c0af175-3134-46fd-9446-66ef0f8b9950","Type":"ContainerStarted","Data":"6dad3d0dfd54d745ca39003174cc3cda55ac9a6dfdc1cec01e7ce6b0fbab7700"} Oct 31 14:56:19.534661 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:19.534630 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks"] Oct 31 14:56:19.883271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:19.883176 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:20.012650 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:20.012613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:20.484223 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:20.484184 4865 generic.go:356] "Generic (PLEG): container finished" podID="3c0af175-3134-46fd-9446-66ef0f8b9950" containerID="1dcde198efee65f9b74c1c8977e43a6e7af4c168315f56d2f97bbc3d9ad6d287" exitCode=0 Oct 31 14:56:20.484400 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:20.484266 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" event={"ID":"3c0af175-3134-46fd-9446-66ef0f8b9950","Type":"ContainerDied","Data":"1dcde198efee65f9b74c1c8977e43a6e7af4c168315f56d2f97bbc3d9ad6d287"} Oct 31 14:56:21.627640 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.627617 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:21.674671 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.674643 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-openshift-ca\") pod \"3c0af175-3134-46fd-9446-66ef0f8b9950\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " Oct 31 14:56:21.674796 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.674739 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0af175-3134-46fd-9446-66ef0f8b9950-kube-api-access-rhqb6\" (UniqueName: \"kubernetes.io/projected/3c0af175-3134-46fd-9446-66ef0f8b9950-kube-api-access-rhqb6\") pod \"3c0af175-3134-46fd-9446-66ef0f8b9950\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " Oct 31 14:56:21.674849 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.674804 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-xks-ca\") pod \"3c0af175-3134-46fd-9446-66ef0f8b9950\" (UID: \"3c0af175-3134-46fd-9446-66ef0f8b9950\") " Oct 31 14:56:21.674986 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.674964 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-openshift-ca" pod "3c0af175-3134-46fd-9446-66ef0f8b9950" (UID: "3c0af175-3134-46fd-9446-66ef0f8b9950"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:21.676980 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.676957 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0af175-3134-46fd-9446-66ef0f8b9950-kube-api-access-rhqb6" pod "3c0af175-3134-46fd-9446-66ef0f8b9950" (UID: "3c0af175-3134-46fd-9446-66ef0f8b9950"). InnerVolumeSpecName "kube-api-access-rhqb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:21.677082 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.676985 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-xks-ca" pod "3c0af175-3134-46fd-9446-66ef0f8b9950" (UID: "3c0af175-3134-46fd-9446-66ef0f8b9950"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:56:21.776086 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.776014 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:21.776086 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.776041 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqb6\" (UniqueName: \"kubernetes.io/projected/3c0af175-3134-46fd-9446-66ef0f8b9950-kube-api-access-rhqb6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:21.776086 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:21.776051 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0af175-3134-46fd-9446-66ef0f8b9950-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:22.142685 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:22.142604 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk"] Oct 31 14:56:22.495700 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:22.495665 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:22.498077 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:22.498041 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" event={"ID":"3c0af175-3134-46fd-9446-66ef0f8b9950","Type":"ContainerDied","Data":"6dad3d0dfd54d745ca39003174cc3cda55ac9a6dfdc1cec01e7ce6b0fbab7700"} Oct 31 14:56:22.498198 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:22.498096 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dad3d0dfd54d745ca39003174cc3cda55ac9a6dfdc1cec01e7ce6b0fbab7700" Oct 31 14:56:22.498198 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:22.498121 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl" Oct 31 14:56:22.628737 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:22.628704 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:23.300344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:23.300314 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:24.535544 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:24.535455 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" podUID="0febb63a-d8c7-48f7-8659-bf5da5a7ad81" containerName="payload-processing" containerID="cri-o://ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738" gracePeriod=25 Oct 31 14:56:24.786154 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:24.786099 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:56:24.931900 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:24.931869 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-plugins-config-volume\") pod \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " Oct 31 14:56:24.932082 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:24.932065 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-kube-api-access-gq9ll\" (UniqueName: \"kubernetes.io/projected/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-kube-api-access-gq9ll\") pod \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\" (UID: \"0febb63a-d8c7-48f7-8659-bf5da5a7ad81\") " Oct 31 14:56:24.932281 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:24.932261 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-plugins-config-volume" pod "0febb63a-d8c7-48f7-8659-bf5da5a7ad81" (UID: "0febb63a-d8c7-48f7-8659-bf5da5a7ad81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:24.932391 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:24.932377 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:24.934314 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:24.934280 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-kube-api-access-gq9ll" pod "0febb63a-d8c7-48f7-8659-bf5da5a7ad81" (UID: "0febb63a-d8c7-48f7-8659-bf5da5a7ad81"). InnerVolumeSpecName "kube-api-access-gq9ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:25.033181 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.033150 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9ll\" (UniqueName: \"kubernetes.io/projected/0febb63a-d8c7-48f7-8659-bf5da5a7ad81-kube-api-access-gq9ll\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:25.514290 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.514257 4865 generic.go:356] "Generic (PLEG): container finished" podID="0febb63a-d8c7-48f7-8659-bf5da5a7ad81" containerID="ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738" exitCode=0 Oct 31 14:56:25.514502 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.514307 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" Oct 31 14:56:25.514502 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.514339 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" event={"ID":"0febb63a-d8c7-48f7-8659-bf5da5a7ad81","Type":"ContainerDied","Data":"ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738"} Oct 31 14:56:25.514502 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.514395 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks" event={"ID":"0febb63a-d8c7-48f7-8659-bf5da5a7ad81","Type":"ContainerDied","Data":"ba149e0d93cd8a19441c18d1ce0c5a46b65c42c1e38deb8667bd2a5f2c1c9ff0"} Oct 31 14:56:25.514502 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.514423 4865 scope.go:122] "RemoveContainer" containerID="ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738" Oct 31 14:56:25.523964 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.523945 4865 scope.go:122] "RemoveContainer" containerID="ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738" Oct 31 14:56:25.524202 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:25.524186 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738\": container with ID starting with ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738 not found: ID does not exist" containerID="ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738" Oct 31 14:56:25.524270 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.524209 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738"} err="failed to get container status \"ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738\": rpc error: code = NotFound desc = could not find container \"ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738\": container with ID starting with ed1b3bb61ba45344ef705cc746e03452897662d924f1f0fb05f1bf38a7735738 not found: ID does not exist" Oct 31 14:56:25.680315 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.680282 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks"] Oct 31 14:56:25.731332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:25.731304 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33bc513e-7557489b45-fj8ks"] Oct 31 14:56:26.495292 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:26.495252 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0febb63a-d8c7-48f7-8659-bf5da5a7ad81" path="/var/lib/kubelet/pods/0febb63a-d8c7-48f7-8659-bf5da5a7ad81/volumes" Oct 31 14:56:27.142891 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.142837 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" podUID="ac2dba94-8927-487b-9bcf-436713422469" containerName="payload-processing" containerID="cri-o://ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3" gracePeriod=25 Oct 31 14:56:27.387326 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.387305 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:56:27.457225 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.457124 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2dba94-8927-487b-9bcf-436713422469-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2dba94-8927-487b-9bcf-436713422469-plugins-config-volume\") pod \"ac2dba94-8927-487b-9bcf-436713422469\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " Oct 31 14:56:27.457389 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.457227 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2dba94-8927-487b-9bcf-436713422469-kube-api-access-cj2cf\" (UniqueName: \"kubernetes.io/projected/ac2dba94-8927-487b-9bcf-436713422469-kube-api-access-cj2cf\") pod \"ac2dba94-8927-487b-9bcf-436713422469\" (UID: \"ac2dba94-8927-487b-9bcf-436713422469\") " Oct 31 14:56:27.457639 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.457585 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2dba94-8927-487b-9bcf-436713422469-plugins-config-volume" pod "ac2dba94-8927-487b-9bcf-436713422469" (UID: "ac2dba94-8927-487b-9bcf-436713422469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:27.460140 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.460112 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2dba94-8927-487b-9bcf-436713422469-kube-api-access-cj2cf" pod "ac2dba94-8927-487b-9bcf-436713422469" (UID: "ac2dba94-8927-487b-9bcf-436713422469"). InnerVolumeSpecName "kube-api-access-cj2cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:27.527308 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.527273 4865 generic.go:356] "Generic (PLEG): container finished" podID="ac2dba94-8927-487b-9bcf-436713422469" containerID="ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3" exitCode=0 Oct 31 14:56:27.527504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.527314 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" Oct 31 14:56:27.527504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.527318 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" event={"ID":"ac2dba94-8927-487b-9bcf-436713422469","Type":"ContainerDied","Data":"ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3"} Oct 31 14:56:27.527504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.527376 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk" event={"ID":"ac2dba94-8927-487b-9bcf-436713422469","Type":"ContainerDied","Data":"93685e4b5c4565afb12333ed60a52e79f07b589aae669e6e5ed0dc0cacbaf3bf"} Oct 31 14:56:27.527504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.527408 4865 scope.go:122] "RemoveContainer" containerID="ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3" Oct 31 14:56:27.537415 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.537400 4865 scope.go:122] "RemoveContainer" containerID="ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3" Oct 31 14:56:27.537688 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:27.537669 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3\": container with ID starting with ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3 not found: ID does not exist" containerID="ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3" Oct 31 14:56:27.537733 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.537698 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3"} err="failed to get container status \"ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3\": rpc error: code = NotFound desc = could not find container \"ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3\": container with ID starting with ae420afb181f189d20e85f5f106cf2543b0ed7ed4f4b124c10a9fe0dc032edd3 not found: ID does not exist" Oct 31 14:56:27.544525 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.544504 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:27.558816 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.558798 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2cf\" (UniqueName: \"kubernetes.io/projected/ac2dba94-8927-487b-9bcf-436713422469-kube-api-access-cj2cf\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:27.558891 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.558818 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2dba94-8927-487b-9bcf-436713422469-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:27.699172 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.699137 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk"] Oct 31 14:56:27.703190 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:27.703149 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b671479d-7bdd6489ff-wngqk"] Oct 31 14:56:28.495132 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:28.495105 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2dba94-8927-487b-9bcf-436713422469" path="/var/lib/kubelet/pods/ac2dba94-8927-487b-9bcf-436713422469/volumes" Oct 31 14:56:28.678065 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:28.678034 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:29.312340 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:29.312305 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:29.627778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:29.627699 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:29.802922 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:29.802890 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn"] Oct 31 14:56:29.855575 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:29.855544 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766"] Oct 31 14:56:29.858286 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:29.858255 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33bc513e-jz766"] Oct 31 14:56:29.923074 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:29.922979 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd"] Oct 31 14:56:29.944870 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:29.944843 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:30.086820 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.086089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:30.228214 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.228168 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:30.363595 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.363562 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:30.494958 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.494890 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2b46f3-74f4-4e65-88c5-e97dca098742" path="/var/lib/kubelet/pods/9c2b46f3-74f4-4e65-88c5-e97dca098742/volumes" Oct 31 14:56:30.691823 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.691790 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr"] Oct 31 14:56:30.692401 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.692381 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0febb63a-d8c7-48f7-8659-bf5da5a7ad81" containerName="payload-processing" Oct 31 14:56:30.692528 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.692403 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0febb63a-d8c7-48f7-8659-bf5da5a7ad81" containerName="payload-processing" Oct 31 14:56:30.692528 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.692449 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ac2dba94-8927-487b-9bcf-436713422469" containerName="payload-processing" Oct 31 14:56:30.692528 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.692458 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2dba94-8927-487b-9bcf-436713422469" containerName="payload-processing" Oct 31 14:56:30.697404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.695862 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3c0af175-3134-46fd-9446-66ef0f8b9950" containerName="revoke-keys" Oct 31 14:56:30.697404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.695889 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0af175-3134-46fd-9446-66ef0f8b9950" containerName="revoke-keys" Oct 31 14:56:30.697404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.696046 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2dba94-8927-487b-9bcf-436713422469" containerName="payload-processing" Oct 31 14:56:30.697404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.696062 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0af175-3134-46fd-9446-66ef0f8b9950" containerName="revoke-keys" Oct 31 14:56:30.697404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.696075 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0febb63a-d8c7-48f7-8659-bf5da5a7ad81" containerName="payload-processing" Oct 31 14:56:30.711748 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.711724 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.720660 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.720631 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:30.749822 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.749716 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr"] Oct 31 14:56:30.765646 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.765445 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:56:30.765646 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.765501 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:56:30.765646 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.765534 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:56:30.765646 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.765666 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 14:56:30.800644 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.800612 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.800821 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.800661 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.800821 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.800680 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5jh\" (UniqueName: \"kubernetes.io/projected/2533a41a-c453-447e-a5a2-f2a4f07fad72-kube-api-access-8r5jh\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.902071 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.902035 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.902071 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.902068 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5jh\" (UniqueName: \"kubernetes.io/projected/2533a41a-c453-447e-a5a2-f2a4f07fad72-kube-api-access-8r5jh\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.902284 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.902165 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.902732 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.902711 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.904575 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.904553 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:30.960019 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:30.959989 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5jh\" (UniqueName: \"kubernetes.io/projected/2533a41a-c453-447e-a5a2-f2a4f07fad72-kube-api-access-8r5jh\") pod \"maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:31.026853 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.026768 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:31.270464 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.270433 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:31.404776 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.404602 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:31.406610 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.406584 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr"] Oct 31 14:56:31.407618 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:31.407591 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2533a41a_c453_447e_a5a2_f2a4f07fad72.slice/crio-c95412d7478a860184b6b23f91adcbd95ffd54e733bfee71f6c9b88344323924 WatchSource:0}: Error finding container c95412d7478a860184b6b23f91adcbd95ffd54e733bfee71f6c9b88344323924: Status 404 returned error can't find the container with id c95412d7478a860184b6b23f91adcbd95ffd54e733bfee71f6c9b88344323924 Oct 31 14:56:31.533696 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.533653 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:31.552683 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.552648 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" event={"ID":"2533a41a-c453-447e-a5a2-f2a4f07fad72","Type":"ContainerStarted","Data":"c95412d7478a860184b6b23f91adcbd95ffd54e733bfee71f6c9b88344323924"} Oct 31 14:56:31.673406 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.673335 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:31.878685 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:31.878652 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:32.538041 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:32.537830 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:32.559667 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:32.559626 4865 generic.go:356] "Generic (PLEG): container finished" podID="2533a41a-c453-447e-a5a2-f2a4f07fad72" containerID="9773d1b422af314cfd13ea09c3510799e0fadf6b1e93026ac02e00b6a464c638" exitCode=0 Oct 31 14:56:32.559827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:32.559716 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" event={"ID":"2533a41a-c453-447e-a5a2-f2a4f07fad72","Type":"ContainerDied","Data":"9773d1b422af314cfd13ea09c3510799e0fadf6b1e93026ac02e00b6a464c638"} Oct 31 14:56:33.201748 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.201713 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:33.716495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.716442 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:33.720964 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.720941 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz"] Oct 31 14:56:33.721401 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.721385 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2533a41a-c453-447e-a5a2-f2a4f07fad72" containerName="revoke-keys" Oct 31 14:56:33.721527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.721405 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2533a41a-c453-447e-a5a2-f2a4f07fad72" containerName="revoke-keys" Oct 31 14:56:33.721593 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.721570 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2533a41a-c453-447e-a5a2-f2a4f07fad72" containerName="revoke-keys" Oct 31 14:56:33.724642 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.724625 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.772942 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-openshift-ca\") pod \"2533a41a-c453-447e-a5a2-f2a4f07fad72\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.772991 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2533a41a-c453-447e-a5a2-f2a4f07fad72-kube-api-access-8r5jh\" (UniqueName: \"kubernetes.io/projected/2533a41a-c453-447e-a5a2-f2a4f07fad72-kube-api-access-8r5jh\") pod \"2533a41a-c453-447e-a5a2-f2a4f07fad72\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773073 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-xks-ca\") pod \"2533a41a-c453-447e-a5a2-f2a4f07fad72\" (UID: \"2533a41a-c453-447e-a5a2-f2a4f07fad72\") " Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773258 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773290 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227v8\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-kube-api-access-227v8\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773331 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773365 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773391 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773414 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773444 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773500 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.773523 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.788497 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.780329 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-56bc96-openshift-default-dockercfg-5qxbd\"" Oct 31 14:56:33.789768 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.787454 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-openshift-ca" pod "2533a41a-c453-447e-a5a2-f2a4f07fad72" (UID: "2533a41a-c453-447e-a5a2-f2a4f07fad72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:33.811970 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.811396 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-xks-ca" pod "2533a41a-c453-447e-a5a2-f2a4f07fad72" (UID: "2533a41a-c453-447e-a5a2-f2a4f07fad72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:56:33.816495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.813021 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2533a41a-c453-447e-a5a2-f2a4f07fad72-kube-api-access-8r5jh" pod "2533a41a-c453-447e-a5a2-f2a4f07fad72" (UID: "2533a41a-c453-447e-a5a2-f2a4f07fad72"). InnerVolumeSpecName "kube-api-access-8r5jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:33.825397 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.825305 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:33.825676 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.825646 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz"] Oct 31 14:56:33.874833 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.874800 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.874839 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227v8\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-kube-api-access-227v8\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.874875 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.874916 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.874943 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875027 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.874965 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875261 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875261 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875250 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875278 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875385 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875365 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.875755 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875730 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.876037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875765 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.876037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875767 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:33.876037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875815 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5jh\" (UniqueName: \"kubernetes.io/projected/2533a41a-c453-447e-a5a2-f2a4f07fad72-kube-api-access-8r5jh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:33.876037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875834 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2533a41a-c453-447e-a5a2-f2a4f07fad72-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:33.876037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875934 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.876037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.875983 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.877790 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.877759 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.877896 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.877840 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.935254 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.935228 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227v8\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-kube-api-access-227v8\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.935736 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.935718 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:33.996981 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:33.996943 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:34.037525 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.037490 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:34.414997 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.414920 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:56:34.461761 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.461733 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz"] Oct 31 14:56:34.462942 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:34.462915 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abf9c3e_ff1e_4127_9f44_da9afef94b65.slice/crio-0692bc6551d245224e451cf54bfb2e50a2935cd3dbf24be1fdc171deb05c1c34 WatchSource:0}: Error finding container 0692bc6551d245224e451cf54bfb2e50a2935cd3dbf24be1fdc171deb05c1c34: Status 404 returned error can't find the container with id 0692bc6551d245224e451cf54bfb2e50a2935cd3dbf24be1fdc171deb05c1c34 Oct 31 14:56:34.465243 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.465211 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:56:34.465323 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.465273 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:56:34.465323 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.465301 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:56:34.573375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.573346 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" Oct 31 14:56:34.573760 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.573375 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr" event={"ID":"2533a41a-c453-447e-a5a2-f2a4f07fad72","Type":"ContainerDied","Data":"c95412d7478a860184b6b23f91adcbd95ffd54e733bfee71f6c9b88344323924"} Oct 31 14:56:34.573760 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.573429 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c95412d7478a860184b6b23f91adcbd95ffd54e733bfee71f6c9b88344323924" Oct 31 14:56:34.575557 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.575531 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" event={"ID":"7abf9c3e-ff1e-4127-9f44-da9afef94b65","Type":"ContainerStarted","Data":"69b2bfdc0012a409cecdff92eae76fa40306c4a1d373f88f6df130f409097875"} Oct 31 14:56:34.575694 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.575581 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" event={"ID":"7abf9c3e-ff1e-4127-9f44-da9afef94b65","Type":"ContainerStarted","Data":"0692bc6551d245224e451cf54bfb2e50a2935cd3dbf24be1fdc171deb05c1c34"} Oct 31 14:56:34.708803 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.708752 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podStartSLOduration=1.708734313 podStartE2EDuration="1.708734313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:34.706752647 +0000 UTC m=+640.326304059" watchObservedRunningTime="2026-10-31 14:56:34.708734313 +0000 UTC m=+640.328285722" Oct 31 14:56:34.803687 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.803654 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" podUID="9b1c04e3-142a-406d-8141-7e2864764cdb" containerName="payload-processing" containerID="cri-o://54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d" gracePeriod=25 Oct 31 14:56:34.904745 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.904598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:34.924005 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:34.923975 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" podUID="85747f29-f7af-42a8-9ef4-7caf1483798c" containerName="payload-pre-processing" containerID="cri-o://e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a" gracePeriod=25 Oct 31 14:56:35.030079 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.030050 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:35.037934 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.037907 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:35.043255 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.043239 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:35.065212 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.065179 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:56:35.086942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.086912 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1c04e3-142a-406d-8141-7e2864764cdb-kube-api-access-str9w\" (UniqueName: \"kubernetes.io/projected/9b1c04e3-142a-406d-8141-7e2864764cdb-kube-api-access-str9w\") pod \"9b1c04e3-142a-406d-8141-7e2864764cdb\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " Oct 31 14:56:35.087103 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.086961 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1c04e3-142a-406d-8141-7e2864764cdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c04e3-142a-406d-8141-7e2864764cdb-plugins-config-volume\") pod \"9b1c04e3-142a-406d-8141-7e2864764cdb\" (UID: \"9b1c04e3-142a-406d-8141-7e2864764cdb\") " Oct 31 14:56:35.087355 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.087326 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1c04e3-142a-406d-8141-7e2864764cdb-plugins-config-volume" pod "9b1c04e3-142a-406d-8141-7e2864764cdb" (UID: "9b1c04e3-142a-406d-8141-7e2864764cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:35.089232 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.089203 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1c04e3-142a-406d-8141-7e2864764cdb-kube-api-access-str9w" pod "9b1c04e3-142a-406d-8141-7e2864764cdb" (UID: "9b1c04e3-142a-406d-8141-7e2864764cdb"). InnerVolumeSpecName "kube-api-access-str9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:35.179044 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.179020 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:56:35.188097 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.188077 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85747f29-f7af-42a8-9ef4-7caf1483798c-kube-api-access-8jpcb\" (UniqueName: \"kubernetes.io/projected/85747f29-f7af-42a8-9ef4-7caf1483798c-kube-api-access-8jpcb\") pod \"85747f29-f7af-42a8-9ef4-7caf1483798c\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " Oct 31 14:56:35.188180 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.188161 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85747f29-f7af-42a8-9ef4-7caf1483798c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85747f29-f7af-42a8-9ef4-7caf1483798c-plugins-config-volume\") pod \"85747f29-f7af-42a8-9ef4-7caf1483798c\" (UID: \"85747f29-f7af-42a8-9ef4-7caf1483798c\") " Oct 31 14:56:35.188563 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.188533 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85747f29-f7af-42a8-9ef4-7caf1483798c-plugins-config-volume" pod "85747f29-f7af-42a8-9ef4-7caf1483798c" (UID: "85747f29-f7af-42a8-9ef4-7caf1483798c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:35.188600 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.188587 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b1c04e3-142a-406d-8141-7e2864764cdb-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.188641 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.188599 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-str9w\" (UniqueName: \"kubernetes.io/projected/9b1c04e3-142a-406d-8141-7e2864764cdb-kube-api-access-str9w\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.190089 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.190065 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85747f29-f7af-42a8-9ef4-7caf1483798c-kube-api-access-8jpcb" pod "85747f29-f7af-42a8-9ef4-7caf1483798c" (UID: "85747f29-f7af-42a8-9ef4-7caf1483798c"). InnerVolumeSpecName "kube-api-access-8jpcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:35.289203 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.289139 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85747f29-f7af-42a8-9ef4-7caf1483798c-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.289203 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.289164 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpcb\" (UniqueName: \"kubernetes.io/projected/85747f29-f7af-42a8-9ef4-7caf1483798c-kube-api-access-8jpcb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.460302 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.460261 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:35.581841 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.581758 4865 generic.go:356] "Generic (PLEG): container finished" podID="85747f29-f7af-42a8-9ef4-7caf1483798c" containerID="e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a" exitCode=0 Oct 31 14:56:35.581841 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.581827 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" Oct 31 14:56:35.582067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.581838 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" event={"ID":"85747f29-f7af-42a8-9ef4-7caf1483798c","Type":"ContainerDied","Data":"e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a"} Oct 31 14:56:35.582067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.581890 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd" event={"ID":"85747f29-f7af-42a8-9ef4-7caf1483798c","Type":"ContainerDied","Data":"d09d21e6349532c04e55dd8a997da423d819a3b0f02eb1a581097c2b3fe28a45"} Oct 31 14:56:35.582067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.581923 4865 scope.go:122] "RemoveContainer" containerID="e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a" Oct 31 14:56:35.583428 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.583406 4865 generic.go:356] "Generic (PLEG): container finished" podID="9b1c04e3-142a-406d-8141-7e2864764cdb" containerID="54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d" exitCode=0 Oct 31 14:56:35.583541 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.583464 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" Oct 31 14:56:35.583610 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.583492 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" event={"ID":"9b1c04e3-142a-406d-8141-7e2864764cdb","Type":"ContainerDied","Data":"54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d"} Oct 31 14:56:35.583610 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.583585 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn" event={"ID":"9b1c04e3-142a-406d-8141-7e2864764cdb","Type":"ContainerDied","Data":"568855ba326da80e4c8a701e777284d7b4f6502d3c6d5d72fe5e99671e932658"} Oct 31 14:56:35.583964 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.583948 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:35.585276 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.585261 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:56:35.591954 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.591860 4865 scope.go:122] "RemoveContainer" containerID="e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a" Oct 31 14:56:35.592235 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:35.592217 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a\": container with ID starting with e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a not found: ID does not exist" containerID="e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a" Oct 31 14:56:35.592303 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.592243 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a"} err="failed to get container status \"e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a\": rpc error: code = NotFound desc = could not find container \"e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a\": container with ID starting with e7ddf091e989c764a7ead89d6f265c758fe6bbe8ea27525366abe77790956f3a not found: ID does not exist" Oct 31 14:56:35.592303 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.592259 4865 scope.go:122] "RemoveContainer" containerID="54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d" Oct 31 14:56:35.600576 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.600558 4865 scope.go:122] "RemoveContainer" containerID="54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d" Oct 31 14:56:35.600816 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:35.600798 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d\": container with ID starting with 54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d not found: ID does not exist" containerID="54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d" Oct 31 14:56:35.600862 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.600823 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d"} err="failed to get container status \"54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d\": rpc error: code = NotFound desc = could not find container \"54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d\": container with ID starting with 54815c3e65972b31d2c0b228b0ebf6cfd054f377997da4928403dbb08b4b1d6d not found: ID does not exist" Oct 31 14:56:35.691717 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.691686 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:35.819538 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.819508 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:35.875125 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.875050 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn"] Oct 31 14:56:35.881022 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.880996 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b1c2330c-d6b5dff79-7sdxn"] Oct 31 14:56:35.991980 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.991950 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd"] Oct 31 14:56:35.993675 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:35.993657 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b1c2330c-59b67d4c9-v9bxd"] Oct 31 14:56:36.495623 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:36.495593 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85747f29-f7af-42a8-9ef4-7caf1483798c" path="/var/lib/kubelet/pods/85747f29-f7af-42a8-9ef4-7caf1483798c/volumes" Oct 31 14:56:36.496034 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:36.495948 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1c04e3-142a-406d-8141-7e2864764cdb" path="/var/lib/kubelet/pods/9b1c04e3-142a-406d-8141-7e2864764cdb/volumes" Oct 31 14:56:36.724236 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:36.724206 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:37.101085 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:37.101057 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:37.237884 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:37.237850 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:37.287886 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:37.287857 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl"] Oct 31 14:56:37.339616 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:37.339586 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b1c2330c-5kwtl"] Oct 31 14:56:37.348885 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:37.348856 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp"] Oct 31 14:56:37.349107 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:37.349082 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" containerID="cri-o://053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142" gracePeriod=30 Oct 31 14:56:37.395300 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:37.395217 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:38.154169 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:38.154137 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:38.497220 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:38.497182 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0af175-3134-46fd-9446-66ef0f8b9950" path="/var/lib/kubelet/pods/3c0af175-3134-46fd-9446-66ef0f8b9950/volumes" Oct 31 14:56:38.749259 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:38.749062 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:38.877728 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:38.877699 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:39.075365 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.075279 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:39.222317 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.222284 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:39.354233 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.354136 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:39.478364 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.478334 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:39.736425 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.736397 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8"] Oct 31 14:56:39.736843 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.736828 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9b1c04e3-142a-406d-8141-7e2864764cdb" containerName="payload-processing" Oct 31 14:56:39.736892 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.736843 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1c04e3-142a-406d-8141-7e2864764cdb" containerName="payload-processing" Oct 31 14:56:39.736892 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.736858 4865 cpu_manager.go:414] "containerMap: removing container" podUID="85747f29-f7af-42a8-9ef4-7caf1483798c" containerName="payload-pre-processing" Oct 31 14:56:39.736892 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.736863 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85747f29-f7af-42a8-9ef4-7caf1483798c" containerName="payload-pre-processing" Oct 31 14:56:39.736991 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.736934 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="85747f29-f7af-42a8-9ef4-7caf1483798c" containerName="payload-pre-processing" Oct 31 14:56:39.736991 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.736948 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1c04e3-142a-406d-8141-7e2864764cdb" containerName="payload-processing" Oct 31 14:56:39.748596 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.748549 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:39.796060 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.796028 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8"] Oct 31 14:56:39.847229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.847195 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-029d26\"" Oct 31 14:56:39.847426 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.847325 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-029d26-dockercfg-2tgzt\"" Oct 31 14:56:39.857912 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.857874 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37abe39b-a918-4e12-9b7b-59c499928838-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:39.858177 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.858148 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbg6g\" (UniqueName: \"kubernetes.io/projected/37abe39b-a918-4e12-9b7b-59c499928838-kube-api-access-zbg6g\") pod \"payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:39.958727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.958686 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbg6g\" (UniqueName: \"kubernetes.io/projected/37abe39b-a918-4e12-9b7b-59c499928838-kube-api-access-zbg6g\") pod \"payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:39.958899 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.958742 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37abe39b-a918-4e12-9b7b-59c499928838-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:39.959464 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:39.959439 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37abe39b-a918-4e12-9b7b-59c499928838-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:40.017404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:40.017334 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbg6g\" (UniqueName: \"kubernetes.io/projected/37abe39b-a918-4e12-9b7b-59c499928838-kube-api-access-zbg6g\") pod \"payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:40.063727 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:40.063701 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:40.442985 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:40.442961 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8"] Oct 31 14:56:40.444594 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:40.444564 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37abe39b_a918_4e12_9b7b_59c499928838.slice/crio-5afedc8d2be13cc84ae9e6456f95aa99d9de8bd9790f679d763cc270be45a143 WatchSource:0}: Error finding container 5afedc8d2be13cc84ae9e6456f95aa99d9de8bd9790f679d763cc270be45a143: Status 404 returned error can't find the container with id 5afedc8d2be13cc84ae9e6456f95aa99d9de8bd9790f679d763cc270be45a143 Oct 31 14:56:40.633562 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:40.633507 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" event={"ID":"37abe39b-a918-4e12-9b7b-59c499928838","Type":"ContainerStarted","Data":"d3b8f02f454f65f0612ce04ae6e61c4e25bfe03e41876b5fe6f6126b01a985d1"} Oct 31 14:56:40.633562 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:40.633555 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" event={"ID":"37abe39b-a918-4e12-9b7b-59c499928838","Type":"ContainerStarted","Data":"5afedc8d2be13cc84ae9e6456f95aa99d9de8bd9790f679d763cc270be45a143"} Oct 31 14:56:40.633794 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:40.633578 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:40.750500 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:40.750432 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" podStartSLOduration=1.75041918 podStartE2EDuration="1.75041918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:40.748939123 +0000 UTC m=+646.368490533" watchObservedRunningTime="2026-10-31 14:56:40.75041918 +0000 UTC m=+646.369970590" Oct 31 14:56:42.458688 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.458647 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:42.614716 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.614655 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:56:42.646666 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.646641 4865 generic.go:356] "Generic (PLEG): container finished" podID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerID="053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142" exitCode=0 Oct 31 14:56:42.646805 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.646686 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" Oct 31 14:56:42.646805 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.646726 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" event={"ID":"858acbb3-a143-4c16-a03e-cd46cfeb4137","Type":"ContainerDied","Data":"053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142"} Oct 31 14:56:42.646805 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.646785 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp" event={"ID":"858acbb3-a143-4c16-a03e-cd46cfeb4137","Type":"ContainerDied","Data":"d44cd9334e5ac1e87fe8a7e0585b94ad88b7817b2e97b14ae214d6d52f161e0d"} Oct 31 14:56:42.646965 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.646813 4865 scope.go:122] "RemoveContainer" containerID="053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142" Oct 31 14:56:42.655630 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.655614 4865 scope.go:122] "RemoveContainer" containerID="19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9" Oct 31 14:56:42.663516 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.663499 4865 scope.go:122] "RemoveContainer" containerID="053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142" Oct 31 14:56:42.663781 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:42.663765 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142\": container with ID starting with 053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142 not found: ID does not exist" containerID="053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142" Oct 31 14:56:42.663865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.663794 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142"} err="failed to get container status \"053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142\": rpc error: code = NotFound desc = could not find container \"053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142\": container with ID starting with 053b12b8313150b461e9c472bd482f7f1b705a7a49731af0b0576f434a4a1142 not found: ID does not exist" Oct 31 14:56:42.663865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.663819 4865 scope.go:122] "RemoveContainer" containerID="19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9" Oct 31 14:56:42.664039 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:42.664021 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9\": container with ID starting with 19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9 not found: ID does not exist" containerID="19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9" Oct 31 14:56:42.664078 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.664047 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9"} err="failed to get container status \"19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9\": rpc error: code = NotFound desc = could not find container \"19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9\": container with ID starting with 19719e26e344e1f73f688e6e1aaf188998bd3b8d3e1beb3e98868475335aead9 not found: ID does not exist" Oct 31 14:56:42.695709 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.695681 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-token\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-token\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.695843 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.695740 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-envoy\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.695843 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.695773 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-certs\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.695843 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.695821 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858acbb3-a143-4c16-a03e-cd46cfeb4137-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858acbb3-a143-4c16-a03e-cd46cfeb4137-istiod-ca-cert\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.695995 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.695855 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-credential-socket\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.695995 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.695920 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-podinfo\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.696093 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696021 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-socket\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.696093 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696066 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-data\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.696093 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696074 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-certs" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:42.696242 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696099 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-kube-api-access-8sx86\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-kube-api-access-8sx86\") pod \"858acbb3-a143-4c16-a03e-cd46cfeb4137\" (UID: \"858acbb3-a143-4c16-a03e-cd46cfeb4137\") " Oct 31 14:56:42.696242 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696170 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-credential-socket" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:42.696242 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696202 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858acbb3-a143-4c16-a03e-cd46cfeb4137-istiod-ca-cert" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:42.696399 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696287 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-socket" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:42.696399 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696367 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-data" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:42.696549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696534 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.696603 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696552 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.696603 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696561 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.696603 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696575 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858acbb3-a143-4c16-a03e-cd46cfeb4137-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.696603 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.696586 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.698021 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.698002 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-token" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:42.698289 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.698267 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-kube-api-access-8sx86" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "kube-api-access-8sx86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:42.698547 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.698533 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-envoy" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:42.698637 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.698615 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-podinfo" pod "858acbb3-a143-4c16-a03e-cd46cfeb4137" (UID: "858acbb3-a143-4c16-a03e-cd46cfeb4137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:56:42.747039 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.747012 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr"] Oct 31 14:56:42.798087 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.798055 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sx86\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-kube-api-access-8sx86\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.798225 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.798091 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.798225 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.798107 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.798225 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.798121 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858acbb3-a143-4c16-a03e-cd46cfeb4137-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:42.851060 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:42.851029 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh"] Oct 31 14:56:43.102400 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:43.102371 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:43.196368 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:43.196321 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp"] Oct 31 14:56:43.199460 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:43.199435 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b1c2330c-gw-openshift-default-544fb44c49-2xmwp"] Oct 31 14:56:43.231004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:43.230972 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:44.122194 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:44.122162 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:44.272041 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:44.272010 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:44.516157 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:44.516113 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" path="/var/lib/kubelet/pods/858acbb3-a143-4c16-a03e-cd46cfeb4137/volumes" Oct 31 14:56:45.208151 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:45.208118 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc"] Oct 31 14:56:45.551522 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:45.551424 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:45.676873 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:45.676832 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:47.488551 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:47.488515 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:56:47.493950 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:47.493926 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:47.747318 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:47.747280 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" podUID="46ff68b4-004d-4e45-958d-0d3fc91612ac" containerName="payload-processing" containerID="cri-o://85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862" gracePeriod=25 Oct 31 14:56:47.852122 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:47.852083 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" podUID="a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" containerName="payload-pre-processing" containerID="cri-o://f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223" gracePeriod=25 Oct 31 14:56:47.951092 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:47.949430 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:48.013452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.013394 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:56:48.073186 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.073148 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ff68b4-004d-4e45-958d-0d3fc91612ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ff68b4-004d-4e45-958d-0d3fc91612ac-plugins-config-volume\") pod \"46ff68b4-004d-4e45-958d-0d3fc91612ac\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " Oct 31 14:56:48.073326 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.073293 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ff68b4-004d-4e45-958d-0d3fc91612ac-kube-api-access-cn2nw\" (UniqueName: \"kubernetes.io/projected/46ff68b4-004d-4e45-958d-0d3fc91612ac-kube-api-access-cn2nw\") pod \"46ff68b4-004d-4e45-958d-0d3fc91612ac\" (UID: \"46ff68b4-004d-4e45-958d-0d3fc91612ac\") " Oct 31 14:56:48.073534 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.073510 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ff68b4-004d-4e45-958d-0d3fc91612ac-plugins-config-volume" pod "46ff68b4-004d-4e45-958d-0d3fc91612ac" (UID: "46ff68b4-004d-4e45-958d-0d3fc91612ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:48.073687 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.073666 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ff68b4-004d-4e45-958d-0d3fc91612ac-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:48.076563 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.076537 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ff68b4-004d-4e45-958d-0d3fc91612ac-kube-api-access-cn2nw" pod "46ff68b4-004d-4e45-958d-0d3fc91612ac" (UID: "46ff68b4-004d-4e45-958d-0d3fc91612ac"). InnerVolumeSpecName "kube-api-access-cn2nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:48.096372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.096353 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:56:48.174150 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.174119 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-plugins-config-volume\") pod \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " Oct 31 14:56:48.174339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.174224 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-kube-api-access-dss2n\" (UniqueName: \"kubernetes.io/projected/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-kube-api-access-dss2n\") pod \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\" (UID: \"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4\") " Oct 31 14:56:48.174452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.174438 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn2nw\" (UniqueName: \"kubernetes.io/projected/46ff68b4-004d-4e45-958d-0d3fc91612ac-kube-api-access-cn2nw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:48.174591 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.174567 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-plugins-config-volume" pod "a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" (UID: "a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:48.176320 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.176302 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-kube-api-access-dss2n" pod "a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" (UID: "a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4"). InnerVolumeSpecName "kube-api-access-dss2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:48.275322 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.275250 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:48.275322 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.275277 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dss2n\" (UniqueName: \"kubernetes.io/projected/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4-kube-api-access-dss2n\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:48.680762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.680685 4865 generic.go:356] "Generic (PLEG): container finished" podID="a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" containerID="f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223" exitCode=0 Oct 31 14:56:48.680762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.680744 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" Oct 31 14:56:48.681265 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.680771 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" event={"ID":"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4","Type":"ContainerDied","Data":"f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223"} Oct 31 14:56:48.681265 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.680850 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh" event={"ID":"a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4","Type":"ContainerDied","Data":"493af71b8acea25bc99b499820d6f7dc17ae9b10c82aaafe97c0bf420711a5f8"} Oct 31 14:56:48.681265 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.680877 4865 scope.go:122] "RemoveContainer" containerID="f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223" Oct 31 14:56:48.682599 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.682579 4865 generic.go:356] "Generic (PLEG): container finished" podID="46ff68b4-004d-4e45-958d-0d3fc91612ac" containerID="85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862" exitCode=0 Oct 31 14:56:48.682719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.682632 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" event={"ID":"46ff68b4-004d-4e45-958d-0d3fc91612ac","Type":"ContainerDied","Data":"85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862"} Oct 31 14:56:48.682719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.682642 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" Oct 31 14:56:48.682719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.682675 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr" event={"ID":"46ff68b4-004d-4e45-958d-0d3fc91612ac","Type":"ContainerDied","Data":"af8250052a9e6cb5cad1fff76253bb44c799825da43b9ab3a4c81b938cf11b73"} Oct 31 14:56:48.690011 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.689995 4865 scope.go:122] "RemoveContainer" containerID="f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223" Oct 31 14:56:48.690261 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:48.690242 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223\": container with ID starting with f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223 not found: ID does not exist" containerID="f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223" Oct 31 14:56:48.690316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.690267 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223"} err="failed to get container status \"f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223\": rpc error: code = NotFound desc = could not find container \"f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223\": container with ID starting with f4f59d8e1cca3f24c54ca97954f63bcf09676ad9ada6ac5d05bc9bdca982c223 not found: ID does not exist" Oct 31 14:56:48.690316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.690282 4865 scope.go:122] "RemoveContainer" containerID="85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862" Oct 31 14:56:48.698008 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.697991 4865 scope.go:122] "RemoveContainer" containerID="85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862" Oct 31 14:56:48.698283 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:48.698264 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862\": container with ID starting with 85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862 not found: ID does not exist" containerID="85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862" Oct 31 14:56:48.698352 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.698292 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862"} err="failed to get container status \"85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862\": rpc error: code = NotFound desc = could not find container \"85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862\": container with ID starting with 85df272948fea48b7d142bffafa6eb47a69c2b1dca38d5956f17080f71abd862 not found: ID does not exist" Oct 31 14:56:48.848096 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.848068 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr"] Oct 31 14:56:48.899205 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.899177 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-08b12cc9-7c64685f99-5hcjr"] Oct 31 14:56:48.964850 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.964822 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh"] Oct 31 14:56:48.966539 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.966514 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-08b12cc9-b97464cb7-b5rfh"] Oct 31 14:56:48.970010 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:48.969990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:49.116359 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.116317 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:49.237129 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.237021 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:49.384796 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.384757 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj"] Oct 31 14:56:49.385281 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385262 4865 cpu_manager.go:414] "containerMap: removing container" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" Oct 31 14:56:49.385372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385283 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" Oct 31 14:56:49.385372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385301 4865 cpu_manager.go:414] "containerMap: removing container" podUID="46ff68b4-004d-4e45-958d-0d3fc91612ac" containerName="payload-processing" Oct 31 14:56:49.385372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385310 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ff68b4-004d-4e45-958d-0d3fc91612ac" containerName="payload-processing" Oct 31 14:56:49.385372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385327 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" containerName="payload-pre-processing" Oct 31 14:56:49.385372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385334 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" containerName="payload-pre-processing" Oct 31 14:56:49.385653 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385435 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" Oct 31 14:56:49.385653 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385450 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ff68b4-004d-4e45-958d-0d3fc91612ac" containerName="payload-processing" Oct 31 14:56:49.385653 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385461 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" containerName="payload-pre-processing" Oct 31 14:56:49.385653 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.385489 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" Oct 31 14:56:49.389942 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.389919 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.445171 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.445138 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-56bc96-dockercfg-b89ws\"" Oct 31 14:56:49.445407 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.445159 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-56bc96\"" Oct 31 14:56:49.479540 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.475510 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj"] Oct 31 14:56:49.487495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.487378 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663720c-0379-485b-9af9-7e90dcfc4fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.487495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.487434 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmdk\" (UniqueName: \"kubernetes.io/projected/4663720c-0379-485b-9af9-7e90dcfc4fc7-kube-api-access-ltmdk\") pod \"payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.541177 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.541147 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5"] Oct 31 14:56:49.541605 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.541591 4865 cpu_manager.go:414] "containerMap: removing container" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" Oct 31 14:56:49.541660 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.541607 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858acbb3-a143-4c16-a03e-cd46cfeb4137" containerName="istio-proxy" Oct 31 14:56:49.544749 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.544735 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:49.588639 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.588610 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663720c-0379-485b-9af9-7e90dcfc4fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.588906 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.588660 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmdk\" (UniqueName: \"kubernetes.io/projected/4663720c-0379-485b-9af9-7e90dcfc4fc7-kube-api-access-ltmdk\") pod \"payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.589353 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.589332 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663720c-0379-485b-9af9-7e90dcfc4fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.600204 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.600181 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5"] Oct 31 14:56:49.647499 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.647441 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmdk\" (UniqueName: \"kubernetes.io/projected/4663720c-0379-485b-9af9-7e90dcfc4fc7-kube-api-access-ltmdk\") pod \"payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.689326 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.689299 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0e1dbd-d3a0-4528-900c-529504a0fbed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:49.689865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.689392 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmzv\" (UniqueName: \"kubernetes.io/projected/5f0e1dbd-d3a0-4528-900c-529504a0fbed-kube-api-access-crmzv\") pod \"payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:49.702079 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.702061 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:49.790560 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.790528 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0e1dbd-d3a0-4528-900c-529504a0fbed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:49.790739 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.790593 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmzv\" (UniqueName: \"kubernetes.io/projected/5f0e1dbd-d3a0-4528-900c-529504a0fbed-kube-api-access-crmzv\") pod \"payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:49.791332 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.791309 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0e1dbd-d3a0-4528-900c-529504a0fbed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:49.848861 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.848830 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmzv\" (UniqueName: \"kubernetes.io/projected/5f0e1dbd-d3a0-4528-900c-529504a0fbed-kube-api-access-crmzv\") pod \"payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:49.857490 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:49.857456 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:50.072201 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.072177 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj"] Oct 31 14:56:50.073799 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:50.073769 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4663720c_0379_485b_9af9_7e90dcfc4fc7.slice/crio-f09ea04531ab78c8a5f5c9b27fda90f63a6270515145a66264d521333a94dfcc WatchSource:0}: Error finding container f09ea04531ab78c8a5f5c9b27fda90f63a6270515145a66264d521333a94dfcc: Status 404 returned error can't find the container with id f09ea04531ab78c8a5f5c9b27fda90f63a6270515145a66264d521333a94dfcc Oct 31 14:56:50.208562 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.208519 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" podUID="c53cb03d-60fb-489f-aa69-def6ab054ae7" containerName="payload-processing" containerID="cri-o://3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607" gracePeriod=25 Oct 31 14:56:50.228344 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.228320 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5"] Oct 31 14:56:50.277373 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:50.277341 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0e1dbd_d3a0_4528_900c_529504a0fbed.slice/crio-83d3864b6de69083ea06980555a6edc040a49309f523b1efaeb52b6b39ebdc80 WatchSource:0}: Error finding container 83d3864b6de69083ea06980555a6edc040a49309f523b1efaeb52b6b39ebdc80: Status 404 returned error can't find the container with id 83d3864b6de69083ea06980555a6edc040a49309f523b1efaeb52b6b39ebdc80 Oct 31 14:56:50.453119 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.453096 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:56:50.497604 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.497566 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ff68b4-004d-4e45-958d-0d3fc91612ac" path="/var/lib/kubelet/pods/46ff68b4-004d-4e45-958d-0d3fc91612ac/volumes" Oct 31 14:56:50.497989 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.497968 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4" path="/var/lib/kubelet/pods/a164f69f-11b2-4d3d-b7c8-a57fa1c9a5c4/volumes" Oct 31 14:56:50.598285 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.598197 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53cb03d-60fb-489f-aa69-def6ab054ae7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53cb03d-60fb-489f-aa69-def6ab054ae7-plugins-config-volume\") pod \"c53cb03d-60fb-489f-aa69-def6ab054ae7\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " Oct 31 14:56:50.598457 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.598376 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53cb03d-60fb-489f-aa69-def6ab054ae7-kube-api-access-7w2ds\" (UniqueName: \"kubernetes.io/projected/c53cb03d-60fb-489f-aa69-def6ab054ae7-kube-api-access-7w2ds\") pod \"c53cb03d-60fb-489f-aa69-def6ab054ae7\" (UID: \"c53cb03d-60fb-489f-aa69-def6ab054ae7\") " Oct 31 14:56:50.598630 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.598605 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53cb03d-60fb-489f-aa69-def6ab054ae7-plugins-config-volume" pod "c53cb03d-60fb-489f-aa69-def6ab054ae7" (UID: "c53cb03d-60fb-489f-aa69-def6ab054ae7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:50.598890 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.598870 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53cb03d-60fb-489f-aa69-def6ab054ae7-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:50.600620 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.600595 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53cb03d-60fb-489f-aa69-def6ab054ae7-kube-api-access-7w2ds" pod "c53cb03d-60fb-489f-aa69-def6ab054ae7" (UID: "c53cb03d-60fb-489f-aa69-def6ab054ae7"). InnerVolumeSpecName "kube-api-access-7w2ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:50.701826 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.701792 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w2ds\" (UniqueName: \"kubernetes.io/projected/c53cb03d-60fb-489f-aa69-def6ab054ae7-kube-api-access-7w2ds\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:50.712304 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.712268 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" event={"ID":"5f0e1dbd-d3a0-4528-900c-529504a0fbed","Type":"ContainerStarted","Data":"ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551"} Oct 31 14:56:50.712559 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.712535 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:56:50.712703 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.712582 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" event={"ID":"5f0e1dbd-d3a0-4528-900c-529504a0fbed","Type":"ContainerStarted","Data":"83d3864b6de69083ea06980555a6edc040a49309f523b1efaeb52b6b39ebdc80"} Oct 31 14:56:50.714111 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.714087 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" event={"ID":"4663720c-0379-485b-9af9-7e90dcfc4fc7","Type":"ContainerStarted","Data":"b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c"} Oct 31 14:56:50.714232 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.714134 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" event={"ID":"4663720c-0379-485b-9af9-7e90dcfc4fc7","Type":"ContainerStarted","Data":"f09ea04531ab78c8a5f5c9b27fda90f63a6270515145a66264d521333a94dfcc"} Oct 31 14:56:50.714232 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.714159 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:56:50.715573 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.715551 4865 generic.go:356] "Generic (PLEG): container finished" podID="c53cb03d-60fb-489f-aa69-def6ab054ae7" containerID="3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607" exitCode=0 Oct 31 14:56:50.715675 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.715610 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" Oct 31 14:56:50.715675 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.715635 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" event={"ID":"c53cb03d-60fb-489f-aa69-def6ab054ae7","Type":"ContainerDied","Data":"3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607"} Oct 31 14:56:50.715788 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.715687 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc" event={"ID":"c53cb03d-60fb-489f-aa69-def6ab054ae7","Type":"ContainerDied","Data":"96407c3407e2257988bdd75d868bcc82ecc528cf4cf301391c61d1c2c775e4ab"} Oct 31 14:56:50.715788 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.715715 4865 scope.go:122] "RemoveContainer" containerID="3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607" Oct 31 14:56:50.725278 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.725264 4865 scope.go:122] "RemoveContainer" containerID="3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607" Oct 31 14:56:50.725591 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:56:50.725572 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607\": container with ID starting with 3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607 not found: ID does not exist" containerID="3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607" Oct 31 14:56:50.725664 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.725598 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607"} err="failed to get container status \"3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607\": rpc error: code = NotFound desc = could not find container \"3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607\": container with ID starting with 3569c366e6d2b378cdd79c23e04d13026eee09dfdbe3a57ed1dd0fbe3a382607 not found: ID does not exist" Oct 31 14:56:50.831560 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.831503 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" podStartSLOduration=1.831489403 podStartE2EDuration="1.831489403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:50.830122399 +0000 UTC m=+656.449673810" watchObservedRunningTime="2026-10-31 14:56:50.831489403 +0000 UTC m=+656.451040813" Oct 31 14:56:50.952533 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.949114 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc"] Oct 31 14:56:50.952533 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:50.952132 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ee109dd4-686b9f4bb7-s6nmc"] Oct 31 14:56:51.062546 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.062494 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" podStartSLOduration=2.062454583 podStartE2EDuration="2.062454583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:51.061294175 +0000 UTC m=+656.680845587" watchObservedRunningTime="2026-10-31 14:56:51.062454583 +0000 UTC m=+656.682005993" Oct 31 14:56:51.436662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.436633 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:51.601024 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.600994 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:51.642816 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.642792 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 14:56:51.848745 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.848715 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2"] Oct 31 14:56:51.849163 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.849149 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c53cb03d-60fb-489f-aa69-def6ab054ae7" containerName="payload-processing" Oct 31 14:56:51.849214 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.849165 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53cb03d-60fb-489f-aa69-def6ab054ae7" containerName="payload-processing" Oct 31 14:56:51.849289 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.849280 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53cb03d-60fb-489f-aa69-def6ab054ae7" containerName="payload-processing" Oct 31 14:56:51.853578 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.853562 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:51.907596 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.907569 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-190693-dockercfg-fftvw\"" Oct 31 14:56:51.907862 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.907825 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-190693\"" Oct 31 14:56:51.908657 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.908608 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2"] Oct 31 14:56:51.932860 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.923356 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4bzn\" (UniqueName: \"kubernetes.io/projected/65424c96-2c96-40c4-b781-ac6ace97bf35-kube-api-access-j4bzn\") pod \"payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:51.936416 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:51.936368 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65424c96-2c96-40c4-b781-ac6ace97bf35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:52.006385 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.006355 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz"] Oct 31 14:56:52.010173 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.010145 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.037915 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.037888 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4bzn\" (UniqueName: \"kubernetes.io/projected/65424c96-2c96-40c4-b781-ac6ace97bf35-kube-api-access-j4bzn\") pod \"payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:52.038093 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.038071 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65424c96-2c96-40c4-b781-ac6ace97bf35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:52.038748 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.038731 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65424c96-2c96-40c4-b781-ac6ace97bf35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:52.066510 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.066482 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz"] Oct 31 14:56:52.095937 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.095904 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4bzn\" (UniqueName: \"kubernetes.io/projected/65424c96-2c96-40c4-b781-ac6ace97bf35-kube-api-access-j4bzn\") pod \"payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:52.139401 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.139331 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75q5\" (UniqueName: \"kubernetes.io/projected/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-kube-api-access-v75q5\") pod \"payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.139401 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.139389 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.153710 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.153686 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:52.166607 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.166582 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:52.246253 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.240514 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75q5\" (UniqueName: \"kubernetes.io/projected/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-kube-api-access-v75q5\") pod \"payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.246253 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.240594 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.246253 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.245924 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.304046 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.304016 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75q5\" (UniqueName: \"kubernetes.io/projected/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-kube-api-access-v75q5\") pod \"payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.306582 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.306550 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:52.323439 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.323418 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:52.428991 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.428961 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:52.495740 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.495712 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53cb03d-60fb-489f-aa69-def6ab054ae7" path="/var/lib/kubelet/pods/c53cb03d-60fb-489f-aa69-def6ab054ae7/volumes" Oct 31 14:56:52.558635 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.558616 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2"] Oct 31 14:56:52.560103 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:52.560082 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65424c96_2c96_40c4_b781_ac6ace97bf35.slice/crio-7c4e5749da22cedc07046c6f329a270c928962445b73369f1df12a1da94436d6 WatchSource:0}: Error finding container 7c4e5749da22cedc07046c6f329a270c928962445b73369f1df12a1da94436d6: Status 404 returned error can't find the container with id 7c4e5749da22cedc07046c6f329a270c928962445b73369f1df12a1da94436d6 Oct 31 14:56:52.710243 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.710219 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz"] Oct 31 14:56:52.711191 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:56:52.711169 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9b1a22_b5ce_4118_b9e8_64fb9b8e4b41.slice/crio-68169d60159be4707bcd9101080b8a57eb6fe068d19adf2da750bdea8fac295f WatchSource:0}: Error finding container 68169d60159be4707bcd9101080b8a57eb6fe068d19adf2da750bdea8fac295f: Status 404 returned error can't find the container with id 68169d60159be4707bcd9101080b8a57eb6fe068d19adf2da750bdea8fac295f Oct 31 14:56:52.736879 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.736848 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" event={"ID":"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41","Type":"ContainerStarted","Data":"68169d60159be4707bcd9101080b8a57eb6fe068d19adf2da750bdea8fac295f"} Oct 31 14:56:52.738504 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.738456 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" event={"ID":"65424c96-2c96-40c4-b781-ac6ace97bf35","Type":"ContainerStarted","Data":"a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f"} Oct 31 14:56:52.738580 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.738523 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" event={"ID":"65424c96-2c96-40c4-b781-ac6ace97bf35","Type":"ContainerStarted","Data":"7c4e5749da22cedc07046c6f329a270c928962445b73369f1df12a1da94436d6"} Oct 31 14:56:52.738580 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.738554 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:56:52.855320 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:52.855266 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" podStartSLOduration=1.855253845 podStartE2EDuration="1.855253845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:52.853311469 +0000 UTC m=+658.472862879" watchObservedRunningTime="2026-10-31 14:56:52.855253845 +0000 UTC m=+658.474805252" Oct 31 14:56:53.748405 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:53.748276 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" event={"ID":"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41","Type":"ContainerStarted","Data":"3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7"} Oct 31 14:56:53.748405 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:53.748334 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:56:53.865175 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:53.865113 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" podStartSLOduration=2.865100582 podStartE2EDuration="2.865100582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:53.863914652 +0000 UTC m=+659.483466064" watchObservedRunningTime="2026-10-31 14:56:53.865100582 +0000 UTC m=+659.484651993" Oct 31 14:56:56.949045 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:56.949012 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:57.104336 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:57.104307 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:56:57.241959 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:56:57.241799 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:00.240506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:00.240460 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:57:01.270055 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:01.270022 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:01.734165 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:01.734141 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 14:57:01.734339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:01.734203 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 14:57:02.804659 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:02.804626 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:02.978006 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:02.977975 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:03.112918 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:03.112839 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:03.242877 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:03.242839 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:03.751911 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:03.751880 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 14:57:04.755069 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:04.755040 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 14:57:04.784552 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:04.784516 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:04.787017 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:04.786986 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx"] Oct 31 14:57:04.787303 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:04.787275 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" containerID="cri-o://18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e" gracePeriod=30 Oct 31 14:57:05.568274 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:05.568175 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:06.080058 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.080029 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m"] Oct 31 14:57:06.085506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.085484 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.167004 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.166976 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:06.190574 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.190541 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m"] Oct 31 14:57:06.190772 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.190736 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-57815b-openshift-default-dockercfg-9dqr9\"" Oct 31 14:57:06.200415 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200387 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde907c-ced4-4f51-9048-8cb571722f91-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200461 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctkw\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-kube-api-access-fctkw\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200530 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde907c-ced4-4f51-9048-8cb571722f91-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200646 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200565 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200694 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200671 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200742 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200726 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200795 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200754 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200836 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200822 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.200867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.200857 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.301875 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.301845 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde907c-ced4-4f51-9048-8cb571722f91-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.301889 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.301926 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.301970 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302149 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302104 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302199 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302163 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302261 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302221 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302322 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302289 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde907c-ced4-4f51-9048-8cb571722f91-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302381 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302331 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctkw\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-kube-api-access-fctkw\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302381 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302346 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302518 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302379 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302590 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302518 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302638 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302622 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.302982 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.302964 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde907c-ced4-4f51-9048-8cb571722f91-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.304523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.304497 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.304523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.304507 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde907c-ced4-4f51-9048-8cb571722f91-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.354392 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.354290 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:06.364917 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.364889 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.365063 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.365024 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctkw\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-kube-api-access-fctkw\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.398998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.398965 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:06.823954 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:57:06.823918 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde907c_ced4_4f51_9048_8cb571722f91.slice/crio-677b81127513c0bd881a70280305776c85632891b994e2f707145e2e6ff153d7 WatchSource:0}: Error finding container 677b81127513c0bd881a70280305776c85632891b994e2f707145e2e6ff153d7: Status 404 returned error can't find the container with id 677b81127513c0bd881a70280305776c85632891b994e2f707145e2e6ff153d7 Oct 31 14:57:06.824578 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.824558 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m"] Oct 31 14:57:06.826375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.826346 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:06.826436 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.826407 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:06.826498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:06.826439 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:07.667271 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:07.667234 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:07.828854 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:07.828821 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" event={"ID":"5bde907c-ced4-4f51-9048-8cb571722f91","Type":"ContainerStarted","Data":"dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54"} Oct 31 14:57:07.829025 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:07.828910 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" event={"ID":"5bde907c-ced4-4f51-9048-8cb571722f91","Type":"ContainerStarted","Data":"677b81127513c0bd881a70280305776c85632891b994e2f707145e2e6ff153d7"} Oct 31 14:57:07.987407 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:07.987355 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podStartSLOduration=1.987340407 podStartE2EDuration="1.987340407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:07.985881418 +0000 UTC m=+673.605432845" watchObservedRunningTime="2026-10-31 14:57:07.987340407 +0000 UTC m=+673.606891818" Oct 31 14:57:08.399383 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:08.399305 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:08.404496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:08.404447 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:08.834074 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:08.834046 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:08.835064 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:08.835046 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:09.314762 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:09.314723 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:09.529907 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:09.529876 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:09.654034 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:09.653956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:09.835593 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:09.835568 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:57:09.911326 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:09.911291 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr"] Oct 31 14:57:09.952783 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:09.952753 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ee109dd4-hxbqr"] Oct 31 14:57:09.953324 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:09.953269 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:10.062833 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.062807 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:57:10.141032 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141002 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-socket\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141227 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141061 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istiod-ca-cert\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141227 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141091 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-envoy\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141227 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141112 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-credential-socket\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141227 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141154 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-token\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-token\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141227 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141181 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-kube-api-access-l4fz7\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-kube-api-access-l4fz7\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141698 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141233 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-data\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141698 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141282 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-certs\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141698 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141309 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-socket" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:10.141698 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141332 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-podinfo\") pod \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\" (UID: \"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b\") " Oct 31 14:57:10.141698 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141519 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-credential-socket" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:10.142173 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141772 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-certs" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:10.142173 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141801 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.142173 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141815 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istiod-ca-cert" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:57:10.142173 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.141823 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.142396 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.142227 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-data" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:10.143996 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.143969 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-envoy" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:10.144687 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.144662 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-kube-api-access-l4fz7" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "kube-api-access-l4fz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:57:10.145604 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.145578 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-token" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:57:10.146015 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.145992 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-podinfo" pod "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" (UID: "fb6f03e0-85c1-4ba2-8c64-98dc0f56130b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:57:10.243255 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.243221 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.243255 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.243248 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.243255 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.243258 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4fz7\" (UniqueName: \"kubernetes.io/projected/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-kube-api-access-l4fz7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.243517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.243267 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.243517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.243276 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.243517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.243285 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.243517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.243293 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:10.496199 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.496165 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2533a41a-c453-447e-a5a2-f2a4f07fad72" path="/var/lib/kubelet/pods/2533a41a-c453-447e-a5a2-f2a4f07fad72/volumes" Oct 31 14:57:10.744373 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.744313 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:10.845454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.845373 4865 generic.go:356] "Generic (PLEG): container finished" podID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerID="18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e" exitCode=0 Oct 31 14:57:10.845454 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.845426 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" Oct 31 14:57:10.845947 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.845457 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" event={"ID":"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b","Type":"ContainerDied","Data":"18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e"} Oct 31 14:57:10.845947 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.845536 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx" event={"ID":"fb6f03e0-85c1-4ba2-8c64-98dc0f56130b","Type":"ContainerDied","Data":"7a3a715cff8c08aa7eed8271014f2154659ca89a25672099972409cf7f193ce3"} Oct 31 14:57:10.845947 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.845558 4865 scope.go:122] "RemoveContainer" containerID="18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e" Oct 31 14:57:10.854917 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.854896 4865 scope.go:122] "RemoveContainer" containerID="280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385" Oct 31 14:57:10.863449 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.863429 4865 scope.go:122] "RemoveContainer" containerID="18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e" Oct 31 14:57:10.863782 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:57:10.863762 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e\": container with ID starting with 18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e not found: ID does not exist" containerID="18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e" Oct 31 14:57:10.863867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.863789 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e"} err="failed to get container status \"18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e\": rpc error: code = NotFound desc = could not find container \"18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e\": container with ID starting with 18cbc93a18cd20f049133f4a50cad7412f4a01624adb91c7123ca307d919242e not found: ID does not exist" Oct 31 14:57:10.863867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.863805 4865 scope.go:122] "RemoveContainer" containerID="280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385" Oct 31 14:57:10.864061 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:57:10.864039 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385\": container with ID starting with 280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385 not found: ID does not exist" containerID="280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385" Oct 31 14:57:10.864123 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:10.864067 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385"} err="failed to get container status \"280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385\": rpc error: code = NotFound desc = could not find container \"280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385\": container with ID starting with 280b8a7accfde2986733cee1565c3fe2d497f65cdf1cbe33e8fa72fd9d1e7385 not found: ID does not exist" Oct 31 14:57:11.082930 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.082899 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx"] Oct 31 14:57:11.112550 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.112049 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-08b12cc9-openshift-default-6f74d45f49-lb6gx"] Oct 31 14:57:11.326792 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.326754 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq"] Oct 31 14:57:11.327258 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.327237 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" Oct 31 14:57:11.327345 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.327259 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" Oct 31 14:57:11.327345 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.327292 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" Oct 31 14:57:11.327345 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.327301 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" Oct 31 14:57:11.327527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.327407 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" Oct 31 14:57:11.327527 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.327422 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" containerName="istio-proxy" Oct 31 14:57:11.331915 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.331893 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.353810 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.353790 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdpwm\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-kube-api-access-cdpwm\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.353905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.353826 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.353905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.353898 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.353989 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.353915 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.353989 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.353935 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.353989 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.353954 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ad1b6a-2eab-443b-93a0-885e77829e2c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.354089 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.354067 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.354128 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.354114 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.354202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.354188 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.422649 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.422558 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2348-openshift-default-dockercfg-k9qcg\"" Oct 31 14:57:11.429152 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.425844 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq"] Oct 31 14:57:11.429997 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.429790 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:11.455043 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455019 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdpwm\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-kube-api-access-cdpwm\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455157 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455055 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455200 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455182 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455233 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455212 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455285 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455237 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455285 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455262 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ad1b6a-2eab-443b-93a0-885e77829e2c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455395 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455298 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455452 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455433 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455530 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455437 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455592 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455548 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455592 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455576 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455749 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455727 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.455891 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455865 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.456014 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.455995 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ad1b6a-2eab-443b-93a0-885e77829e2c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.457656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.457632 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.458213 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.458183 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.546537 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.546457 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.546927 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.546901 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdpwm\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-kube-api-access-cdpwm\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.569889 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.569856 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:11.644657 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.644624 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:11.729481 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:11.729427 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:12.119849 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:12.117574 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq"] Oct 31 14:57:12.120781 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:57:12.120754 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ad1b6a_2eab_443b_93a0_885e77829e2c.slice/crio-81f927a20f2b30f2fb8ef83575d2b333fe3b96fadd7fce063d21bec314a1780d WatchSource:0}: Error finding container 81f927a20f2b30f2fb8ef83575d2b333fe3b96fadd7fce063d21bec314a1780d: Status 404 returned error can't find the container with id 81f927a20f2b30f2fb8ef83575d2b333fe3b96fadd7fce063d21bec314a1780d Oct 31 14:57:12.123291 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:12.123258 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:12.123373 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:12.123324 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:12.123373 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:12.123353 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:12.506493 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:12.500764 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6f03e0-85c1-4ba2-8c64-98dc0f56130b" path="/var/lib/kubelet/pods/fb6f03e0-85c1-4ba2-8c64-98dc0f56130b/volumes" Oct 31 14:57:12.859348 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:12.859232 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" event={"ID":"90ad1b6a-2eab-443b-93a0-885e77829e2c","Type":"ContainerStarted","Data":"72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523"} Oct 31 14:57:12.859348 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:12.859283 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" event={"ID":"90ad1b6a-2eab-443b-93a0-885e77829e2c","Type":"ContainerStarted","Data":"81f927a20f2b30f2fb8ef83575d2b333fe3b96fadd7fce063d21bec314a1780d"} Oct 31 14:57:13.080006 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:13.079954 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podStartSLOduration=2.079940109 podStartE2EDuration="2.079940109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:13.028041412 +0000 UTC m=+678.647592824" watchObservedRunningTime="2026-10-31 14:57:13.079940109 +0000 UTC m=+678.699491520" Oct 31 14:57:13.644935 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:13.644899 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:13.650187 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:13.650162 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:13.864087 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:13.864053 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:13.865018 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:13.865001 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:57:14.720294 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:14.720258 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:14.849032 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:14.849006 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:14.976360 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:14.976280 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:16.154611 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:16.154577 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:16.322063 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:16.322018 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:16.475207 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:16.475172 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:17.492999 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:57:17.492965 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:57:19.582283 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.582246 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:19.752546 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.752513 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:19.897983 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.897881 4865 generic.go:356] "Generic (PLEG): container finished" podID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerID="69b2bfdc0012a409cecdff92eae76fa40306c4a1d373f88f6df130f409097875" exitCode=0 Oct 31 14:57:19.897983 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.897954 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" event={"ID":"7abf9c3e-ff1e-4127-9f44-da9afef94b65","Type":"ContainerDied","Data":"69b2bfdc0012a409cecdff92eae76fa40306c4a1d373f88f6df130f409097875"} Oct 31 14:57:19.898414 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.898392 4865 scope.go:122] "RemoveContainer" containerID="69b2bfdc0012a409cecdff92eae76fa40306c4a1d373f88f6df130f409097875" Oct 31 14:57:19.899925 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.899890 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:19.900028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.899983 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:19.900091 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:19.900032 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:20.114974 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:20.114938 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:20.584433 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:20.584406 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:57:20.818570 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:20.816047 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:20.920137 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:20.920047 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" event={"ID":"7abf9c3e-ff1e-4127-9f44-da9afef94b65","Type":"ContainerStarted","Data":"3be40b659000f55515a8c2441ef66f16762502403a45d18310e860d49b0def1e"} Oct 31 14:57:21.038001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:21.037963 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:57:21.039654 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:21.039627 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 14:57:21.039784 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:21.039681 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 14:57:21.424311 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:21.424278 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:21.568456 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:21.568424 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:21.714550 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:21.714432 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:22.038067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:22.037982 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 14:57:22.038067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:22.038034 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 14:57:23.038462 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:23.038430 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 14:57:23.038928 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:23.038496 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 14:57:24.038779 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:24.038744 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 14:57:24.039197 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:24.038807 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 14:57:25.038342 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:25.038302 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 14:57:25.038546 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:25.038357 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 14:57:25.452353 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:25.452221 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:26.262013 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:26.261981 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:57:26.262972 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:26.262947 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:57:26.264097 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:26.264072 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 14:57:27.624451 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.624413 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm"] Oct 31 14:57:27.629508 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.629464 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.683790 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.683757 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm"] Oct 31 14:57:27.690489 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.690436 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-model-cache\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.690662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.690557 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89brt\" (UniqueName: \"kubernetes.io/projected/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-kube-api-access-89brt\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.690662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.690589 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-home\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.690792 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.690670 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-dshm\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.690792 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.690708 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tls-certs\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.690792 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.690736 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tmp-dir\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.717685 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.717658 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:27.732378 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.732356 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"kube-root-ca.crt\"" Oct 31 14:57:27.732517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.732497 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"openshift-service-ca.crt\"" Oct 31 14:57:27.732517 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.732504 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"default-dockercfg-vznpg\"" Oct 31 14:57:27.734133 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.734116 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"simulator-w2-029d26-kserve-self-signed-certs\"" Oct 31 14:57:27.792293 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.792251 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-model-cache\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.802846 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.802794 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89brt\" (UniqueName: \"kubernetes.io/projected/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-kube-api-access-89brt\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.803797 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.803596 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-home\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.804420 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.804393 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-dshm\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.804939 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.804913 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tls-certs\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.805168 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.805152 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tmp-dir\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.805380 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.799951 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-model-cache\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.805496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.804318 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-home\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.805819 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.805802 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tmp-dir\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.807244 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.807218 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-dshm\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.808502 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.808442 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tls-certs\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.864201 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.864174 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89brt\" (UniqueName: \"kubernetes.io/projected/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-kube-api-access-89brt\") pod \"simulator-w2-029d26-kserve-84868b5d68-mfkzm\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:27.941849 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:27.941763 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:28.029849 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:28.029819 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:28.317552 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:28.317530 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm"] Oct 31 14:57:28.318355 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:57:28.318326 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbb1205e_6a6d_47a6_a09f_63b2f627ac1e.slice/crio-a693416de7ce41bc4ca6e26dc8d2bfdc07de43f515a1e98f9f34b1349d163989 WatchSource:0}: Error finding container a693416de7ce41bc4ca6e26dc8d2bfdc07de43f515a1e98f9f34b1349d163989: Status 404 returned error can't find the container with id a693416de7ce41bc4ca6e26dc8d2bfdc07de43f515a1e98f9f34b1349d163989 Oct 31 14:57:28.630578 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:28.630496 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:28.763108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:28.763076 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:28.892405 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:28.892333 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:28.968707 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:28.968671 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" event={"ID":"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e","Type":"ContainerStarted","Data":"67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77"} Oct 31 14:57:28.968707 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:28.968717 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" event={"ID":"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e","Type":"ContainerStarted","Data":"a693416de7ce41bc4ca6e26dc8d2bfdc07de43f515a1e98f9f34b1349d163989"} Oct 31 14:57:29.088410 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:29.088354 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podStartSLOduration=2.088335859 podStartE2EDuration="2.088335859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:29.087380362 +0000 UTC m=+694.706931761" watchObservedRunningTime="2026-10-31 14:57:29.088335859 +0000 UTC m=+694.707887270" Oct 31 14:57:29.941965 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:29.941924 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:30.452006 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:30.451977 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:30.634915 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:30.634883 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:30.766459 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:30.766424 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:32.523749 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:32.523717 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:57:33.470136 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:33.470094 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:34.077952 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:34.077921 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:34.214312 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:34.214276 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:34.341582 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:34.341500 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:35.892393 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:35.892361 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:37.942327 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:37.942291 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:37.955306 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:37.955282 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:38.027410 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:38.027378 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 14:57:40.858342 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:40.858309 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:41.016080 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:41.016030 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:41.159778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:41.159689 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:45.792352 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:45.792317 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:47.491371 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:57:47.491329 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:57:48.299767 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:48.299720 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:49.328775 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:49.328741 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:49.642240 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:49.642137 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:57:50.081132 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.081096 4865 generic.go:356] "Generic (PLEG): container finished" podID="5bde907c-ced4-4f51-9048-8cb571722f91" containerID="dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54" exitCode=0 Oct 31 14:57:50.081312 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.081170 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" event={"ID":"5bde907c-ced4-4f51-9048-8cb571722f91","Type":"ContainerDied","Data":"dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54"} Oct 31 14:57:50.081628 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.081611 4865 scope.go:122] "RemoveContainer" containerID="dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54" Oct 31 14:57:50.082915 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.082881 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:50.083013 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.082951 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:50.083013 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.082990 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:50.578369 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.578332 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:50.864144 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.864052 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x"] Oct 31 14:57:50.868063 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.868039 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:50.942489 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.932153 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x"] Oct 31 14:57:50.942489 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.932647 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"kube-root-ca.crt\"" Oct 31 14:57:50.957455 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.957414 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:50.972851 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.972822 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"openshift-service-ca.crt\"" Oct 31 14:57:50.973042 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.972944 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"default-dockercfg-zxkjx\"" Oct 31 14:57:50.973116 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.973052 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"simulator-w0-56bc96-kserve-self-signed-certs\"" Oct 31 14:57:50.976514 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.976487 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-model-cache\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:50.976756 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.976737 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352ed02e-2cd6-4d2c-b978-a55077005e59-tls-certs\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:50.976880 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.976864 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpb7\" (UniqueName: \"kubernetes.io/projected/352ed02e-2cd6-4d2c-b978-a55077005e59-kube-api-access-7qpb7\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:50.977011 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.976998 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-tmp-dir\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:50.977112 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.977100 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-home\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:50.977228 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:50.977215 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-dshm\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.079496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.078883 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-tmp-dir\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.079496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.078938 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-tmp-dir\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.079496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.078973 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-home\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.079496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.079021 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-dshm\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.079496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.079158 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-model-cache\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.079496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.079206 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352ed02e-2cd6-4d2c-b978-a55077005e59-tls-certs\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.079496 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.079233 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpb7\" (UniqueName: \"kubernetes.io/projected/352ed02e-2cd6-4d2c-b978-a55077005e59-kube-api-access-7qpb7\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.083026 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.080180 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-model-cache\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.083026 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.080780 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-home\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.085958 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.085919 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-dshm\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.088128 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.088102 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352ed02e-2cd6-4d2c-b978-a55077005e59-tls-certs\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.092697 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.092674 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" event={"ID":"5bde907c-ced4-4f51-9048-8cb571722f91","Type":"ContainerStarted","Data":"35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916"} Oct 31 14:57:51.152208 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.152099 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpb7\" (UniqueName: \"kubernetes.io/projected/352ed02e-2cd6-4d2c-b978-a55077005e59-kube-api-access-7qpb7\") pod \"simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.182543 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.182492 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:51.406009 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.405877 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:51.415001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.414635 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 14:57:51.415001 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.414691 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 14:57:51.636498 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:51.635272 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x"] Oct 31 14:57:51.646506 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:57:51.646430 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod352ed02e_2cd6_4d2c_b978_a55077005e59.slice/crio-38a616039f888cc47360338fa8a07c3bb96dbd16f284f1d77d1924dcbe8c900a WatchSource:0}: Error finding container 38a616039f888cc47360338fa8a07c3bb96dbd16f284f1d77d1924dcbe8c900a: Status 404 returned error can't find the container with id 38a616039f888cc47360338fa8a07c3bb96dbd16f284f1d77d1924dcbe8c900a Oct 31 14:57:52.168394 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:52.168350 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" event={"ID":"352ed02e-2cd6-4d2c-b978-a55077005e59","Type":"ContainerStarted","Data":"7bd6756788eee17b8eddcebc6a90a74d77b2ed47e67f7f56f6b45c835a9fbbe1"} Oct 31 14:57:52.168394 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:52.168400 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" event={"ID":"352ed02e-2cd6-4d2c-b978-a55077005e59","Type":"ContainerStarted","Data":"38a616039f888cc47360338fa8a07c3bb96dbd16f284f1d77d1924dcbe8c900a"} Oct 31 14:57:52.182632 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:52.182608 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:57:52.285493 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:52.285422 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podStartSLOduration=2.285407779 podStartE2EDuration="2.285407779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:52.284058557 +0000 UTC m=+717.903609969" watchObservedRunningTime="2026-10-31 14:57:52.285407779 +0000 UTC m=+717.904959190" Oct 31 14:57:52.403375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:52.402931 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 14:57:52.403375 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:52.402991 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 14:57:53.408061 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:53.408020 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 14:57:53.408751 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:53.408703 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 14:57:53.836607 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:53.836568 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:54.253393 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:54.253351 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:54.402495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:54.401854 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 14:57:54.402495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:54.401912 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 14:57:54.746148 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:54.746102 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:54.916549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:54.915907 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:55.238574 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:55.229555 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:57:55.416491 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:55.403854 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 31 14:57:55.416491 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:55.403916 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 31 14:57:56.459239 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:56.459172 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:56.460817 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:56.460793 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 14:57:58.120587 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:58.120550 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:57:58.876615 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:58.875012 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:55832->10.132.0.63:15021: read: connection reset by peer" start-of-body= Oct 31 14:57:58.876615 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:58.875066 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:55832->10.132.0.63:15021: read: connection reset by peer" Oct 31 14:57:59.205245 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:59.205143 4865 generic.go:356] "Generic (PLEG): container finished" podID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerID="72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523" exitCode=0 Oct 31 14:57:59.205245 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:59.205225 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" event={"ID":"90ad1b6a-2eab-443b-93a0-885e77829e2c","Type":"ContainerDied","Data":"72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523"} Oct 31 14:57:59.205692 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:59.205663 4865 scope.go:122] "RemoveContainer" containerID="72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523" Oct 31 14:57:59.206925 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:59.206899 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:59.206992 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:59.206965 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:59.207028 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:59.207001 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:57:59.913501 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:57:59.903886 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:00.222632 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:00.222598 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" event={"ID":"90ad1b6a-2eab-443b-93a0-885e77829e2c","Type":"ContainerStarted","Data":"0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d"} Oct 31 14:58:00.644869 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:00.644788 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:58:00.645037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:00.644906 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 14:58:00.645037 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:00.644948 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 14:58:01.182956 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:01.182923 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:58:01.220253 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:01.220224 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:58:01.238937 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:01.238915 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 14:58:01.644995 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:01.644961 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 14:58:01.645179 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:01.645025 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 14:58:02.644920 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:02.644883 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 14:58:02.645322 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:02.644932 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 14:58:03.644905 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:03.644874 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 14:58:03.645143 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:03.644963 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 14:58:04.645170 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:04.645136 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 31 14:58:04.645670 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:04.645195 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 31 14:58:04.709161 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:04.709113 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:05.648664 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:05.648636 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:58:05.649120 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:05.649103 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:58:05.649690 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:05.649672 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 14:58:06.474016 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:06.473984 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:07.080820 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:07.080785 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:07.219511 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:07.219455 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:07.353868 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:07.353781 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:09.842904 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:09.842872 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:12.011865 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.011834 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q"] Oct 31 14:58:12.015314 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.015295 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.100171 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.100132 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:12.119885 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.119859 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"premium-w2-029d26-kserve-self-signed-certs\"" Oct 31 14:58:12.121967 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.121949 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q"] Oct 31 14:58:12.164521 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.164482 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-dshm\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.164867 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.164843 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-tmp-dir\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.165046 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.165029 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da378e66-4f18-415c-a334-4aeb5a33ce75-tls-certs\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.165218 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.165204 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-home\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.165609 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.165588 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-model-cache\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.172374 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.171582 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6m4\" (UniqueName: \"kubernetes.io/projected/da378e66-4f18-415c-a334-4aeb5a33ce75-kube-api-access-hj6m4\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.273202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273107 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-dshm\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.273202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273164 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-tmp-dir\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.273202 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273197 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da378e66-4f18-415c-a334-4aeb5a33ce75-tls-certs\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.273506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273235 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-home\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.273506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273269 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-model-cache\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.273506 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273302 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6m4\" (UniqueName: \"kubernetes.io/projected/da378e66-4f18-415c-a334-4aeb5a33ce75-kube-api-access-hj6m4\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.274314 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273802 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-home\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.274314 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.273907 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-model-cache\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.274314 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.274044 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-tmp-dir\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.281911 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.281886 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da378e66-4f18-415c-a334-4aeb5a33ce75-tls-certs\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.282084 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.281907 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-dshm\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.331991 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.331961 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6m4\" (UniqueName: \"kubernetes.io/projected/da378e66-4f18-415c-a334-4aeb5a33ce75-kube-api-access-hj6m4\") pod \"premium-w2-029d26-kserve-94b44467c-mcf4q\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.627366 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.627284 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:12.997591 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:12.997562 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q"] Oct 31 14:58:12.998612 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:58:12.998590 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda378e66_4f18_415c_a334_4aeb5a33ce75.slice/crio-7319779579f661a91a30e925036f9e275c8f1813dd71af16ed700c7d8120bd98 WatchSource:0}: Error finding container 7319779579f661a91a30e925036f9e275c8f1813dd71af16ed700c7d8120bd98: Status 404 returned error can't find the container with id 7319779579f661a91a30e925036f9e275c8f1813dd71af16ed700c7d8120bd98 Oct 31 14:58:13.301728 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:13.301648 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" event={"ID":"da378e66-4f18-415c-a334-4aeb5a33ce75","Type":"ContainerStarted","Data":"68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d"} Oct 31 14:58:13.301728 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:13.301695 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" event={"ID":"da378e66-4f18-415c-a334-4aeb5a33ce75","Type":"ContainerStarted","Data":"7319779579f661a91a30e925036f9e275c8f1813dd71af16ed700c7d8120bd98"} Oct 31 14:58:13.419168 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:13.419112 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podStartSLOduration=2.419100455 podStartE2EDuration="2.419100455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:58:13.418871025 +0000 UTC m=+739.038422429" watchObservedRunningTime="2026-10-31 14:58:13.419100455 +0000 UTC m=+739.038651888" Oct 31 14:58:13.628129 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:13.628055 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:14.499306 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:14.499266 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:16.676242 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:16.676206 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:17.242959 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:17.242931 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:17.377241 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:17.377208 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:17.501838 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:58:17.501758 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:58:17.516910 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:17.516875 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:19.498834 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:19.498798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:19.607840 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:19.607812 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:58:22.628304 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:22.628274 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:22.641051 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:22.641027 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:23.363829 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:23.363799 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 14:58:24.191898 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:24.191863 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:28.895483 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:28.895437 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:30.848547 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:30.848501 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:33.602372 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:33.602327 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:37.985154 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:37.985076 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:58:38.406927 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:38.406781 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:43.113360 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:43.113294 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:45.172837 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:45.172790 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:47.489365 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:58:47.489326 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:58:47.825378 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:47.825344 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:48.642519 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:48.642489 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:49.217289 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.217256 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk"] Oct 31 14:58:49.220966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.220952 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.280770 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.280737 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk4v\" (UniqueName: \"kubernetes.io/projected/105da59e-13ba-48f4-a007-5be0f8203eeb-kube-api-access-sbk4v\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.280939 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.280793 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-model-cache\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.280993 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.280927 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-home\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.280993 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.280973 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105da59e-13ba-48f4-a007-5be0f8203eeb-tls-certs\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.281109 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.281097 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-dshm\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.281145 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.281119 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-tmp-dir\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.297085 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.297052 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:49.326926 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.326898 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"kube-root-ca.crt\"" Oct 31 14:58:49.326926 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.326902 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"openshift-service-ca.crt\"" Oct 31 14:58:49.326926 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.326903 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"default-dockercfg-7gwx9\"" Oct 31 14:58:49.327134 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.327023 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"premium-w3-190693-kserve-self-signed-certs\"" Oct 31 14:58:49.329118 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.329101 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk"] Oct 31 14:58:49.382806 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.382768 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-home\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.383601 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.383452 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-home\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.383753 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.383573 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105da59e-13ba-48f4-a007-5be0f8203eeb-tls-certs\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.383753 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.383739 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-dshm\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.383895 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.383772 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-tmp-dir\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.383895 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.383829 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk4v\" (UniqueName: \"kubernetes.io/projected/105da59e-13ba-48f4-a007-5be0f8203eeb-kube-api-access-sbk4v\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.383895 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.383886 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-model-cache\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.384261 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.384238 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-model-cache\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.384421 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.384319 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-tmp-dir\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.387342 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.387313 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-dshm\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.388708 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.388690 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105da59e-13ba-48f4-a007-5be0f8203eeb-tls-certs\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.443176 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.443142 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk4v\" (UniqueName: \"kubernetes.io/projected/105da59e-13ba-48f4-a007-5be0f8203eeb-kube-api-access-sbk4v\") pod \"premium-w3-190693-kserve-6c766d86dc-49tzk\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.533299 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.533220 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:49.679992 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.679962 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:49.928191 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:49.928169 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk"] Oct 31 14:58:49.929674 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:58:49.929643 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod105da59e_13ba_48f4_a007_5be0f8203eeb.slice/crio-9b4412c50c1c987d4feb327035672b62fcfe5280b14f8619aa25940e789365cf WatchSource:0}: Error finding container 9b4412c50c1c987d4feb327035672b62fcfe5280b14f8619aa25940e789365cf: Status 404 returned error can't find the container with id 9b4412c50c1c987d4feb327035672b62fcfe5280b14f8619aa25940e789365cf Oct 31 14:58:50.503716 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:50.503678 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" event={"ID":"105da59e-13ba-48f4-a007-5be0f8203eeb","Type":"ContainerStarted","Data":"92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e"} Oct 31 14:58:50.503923 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:50.503729 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" event={"ID":"105da59e-13ba-48f4-a007-5be0f8203eeb","Type":"ContainerStarted","Data":"9b4412c50c1c987d4feb327035672b62fcfe5280b14f8619aa25940e789365cf"} Oct 31 14:58:50.533404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:50.533370 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:50.621093 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:50.621045 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podStartSLOduration=1.621032559 podStartE2EDuration="1.621032559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:58:50.620099462 +0000 UTC m=+776.239650873" watchObservedRunningTime="2026-10-31 14:58:50.621032559 +0000 UTC m=+776.240583970" Oct 31 14:58:52.524943 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:52.524911 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:53.569328 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.569298 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2"] Oct 31 14:58:53.573110 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.573094 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.627107 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.627082 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"openshift-service-ca.crt\"" Oct 31 14:58:53.654774 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.654750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:53.678073 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.678053 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"kube-root-ca.crt\"" Oct 31 14:58:53.678264 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.678249 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"default-dockercfg-hc4sm\"" Oct 31 14:58:53.678309 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.678261 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"simulator-w1-fb2348-kserve-self-signed-certs\"" Oct 31 14:58:53.680339 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.680319 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2"] Oct 31 14:58:53.731818 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.731315 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-model-cache\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.731818 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.731356 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tls-certs\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.731818 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.731429 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-home\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.731818 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.731485 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8hr\" (UniqueName: \"kubernetes.io/projected/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-kube-api-access-9l8hr\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.731818 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.731569 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-dshm\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.731818 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.731601 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tmp-dir\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833007 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.832906 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-dshm\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833007 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.832947 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tmp-dir\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833239 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.833031 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-model-cache\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833239 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.833062 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tls-certs\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833239 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.833129 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-home\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833239 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.833163 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8hr\" (UniqueName: \"kubernetes.io/projected/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-kube-api-access-9l8hr\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833448 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.833418 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-model-cache\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833830 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.833808 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-home\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.833830 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.833824 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tmp-dir\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.835435 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.835408 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-dshm\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.835827 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.835807 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tls-certs\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:53.892573 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:53.892543 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8hr\" (UniqueName: \"kubernetes.io/projected/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-kube-api-access-9l8hr\") pod \"simulator-w1-fb2348-kserve-784b4785fc-v4dd2\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:54.185617 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:54.185536 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:54.559436 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:54.559412 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2"] Oct 31 14:58:54.560107 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:58:54.560082 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb29fd0_3b02_45f9_87b9_b919cf25d4bc.slice/crio-02f0497581ce0a319df5240f46addd7962add5f12b58ee6b0b61885260248423 WatchSource:0}: Error finding container 02f0497581ce0a319df5240f46addd7962add5f12b58ee6b0b61885260248423: Status 404 returned error can't find the container with id 02f0497581ce0a319df5240f46addd7962add5f12b58ee6b0b61885260248423 Oct 31 14:58:55.531095 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:55.531055 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" event={"ID":"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc","Type":"ContainerStarted","Data":"bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607"} Oct 31 14:58:55.531495 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:55.531106 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" event={"ID":"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc","Type":"ContainerStarted","Data":"02f0497581ce0a319df5240f46addd7962add5f12b58ee6b0b61885260248423"} Oct 31 14:58:55.648723 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:55.648666 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podStartSLOduration=2.648652024 podStartE2EDuration="2.648652024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:58:55.646867671 +0000 UTC m=+781.266419083" watchObservedRunningTime="2026-10-31 14:58:55.648652024 +0000 UTC m=+781.268203434" Oct 31 14:58:56.186306 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:56.186278 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:58:56.389965 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:56.389931 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:57.216676 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:57.216642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:58:59.533578 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:59.533544 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:59.546837 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:59.546813 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:59.562813 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:59.562788 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 14:58:59.793987 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:58:59.793900 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:59:02.003316 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:02.003282 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:04.185917 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:04.185880 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:59:04.198377 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:04.198352 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:59:04.594920 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:04.594890 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 14:59:06.796680 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:06.796644 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:11.508266 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:11.508228 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:16.214354 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:16.214323 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:17.020215 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:17.020178 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:59:17.490189 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:59:17.490157 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:59:20.920426 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:20.920384 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:23.764838 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:23.764804 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:24.569693 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.569663 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444"] Oct 31 14:59:24.573398 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.573380 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.629350 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.629322 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444"] Oct 31 14:59:24.652417 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.652388 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgl4\" (UniqueName: \"kubernetes.io/projected/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-kube-api-access-mcgl4\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.652565 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.652428 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.652649 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.652563 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.677805 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.677784 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:59:24.677944 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.677927 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 14:59:24.678009 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.677970 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:59:24.678009 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.677990 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:59:24.753212 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.753187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgl4\" (UniqueName: \"kubernetes.io/projected/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-kube-api-access-mcgl4\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.753374 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.753222 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.753374 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.753309 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.754065 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.754001 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.755816 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.755797 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.811324 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.811300 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgl4\" (UniqueName: \"kubernetes.io/projected/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-kube-api-access-mcgl4\") pod \"maas-api-revoke-sub-27a48b23-nn444\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:24.888819 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:24.888721 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:25.266777 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:25.266536 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444"] Oct 31 14:59:25.271138 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:59:25.271109 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd6712d_b0cf_48fc_9fb2_c226efbf92bd.slice/crio-30900af2b3f900b6cbcf53c733954476934e728c04d4a70103d107574559c18f WatchSource:0}: Error finding container 30900af2b3f900b6cbcf53c733954476934e728c04d4a70103d107574559c18f: Status 404 returned error can't find the container with id 30900af2b3f900b6cbcf53c733954476934e728c04d4a70103d107574559c18f Oct 31 14:59:25.598860 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:25.598831 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:25.698907 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:25.698867 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" event={"ID":"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd","Type":"ContainerStarted","Data":"36ff0bee4fb19e909c97d969ae78ebde7e9f5e338b04a3cef198cb317bcc5e50"} Oct 31 14:59:25.699086 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:25.698936 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" event={"ID":"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd","Type":"ContainerStarted","Data":"30900af2b3f900b6cbcf53c733954476934e728c04d4a70103d107574559c18f"} Oct 31 14:59:25.816038 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:25.815989 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" podStartSLOduration=1.815976042 podStartE2EDuration="1.815976042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:25.813779986 +0000 UTC m=+811.433331397" watchObservedRunningTime="2026-10-31 14:59:25.815976042 +0000 UTC m=+811.435527453" Oct 31 14:59:26.705143 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:26.705109 4865 generic.go:356] "Generic (PLEG): container finished" podID="2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" containerID="36ff0bee4fb19e909c97d969ae78ebde7e9f5e338b04a3cef198cb317bcc5e50" exitCode=0 Oct 31 14:59:26.705326 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:26.705184 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" event={"ID":"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd","Type":"ContainerDied","Data":"36ff0bee4fb19e909c97d969ae78ebde7e9f5e338b04a3cef198cb317bcc5e50"} Oct 31 14:59:27.849781 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.849757 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:27.993817 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.993733 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-kube-api-access-mcgl4\" (UniqueName: \"kubernetes.io/projected/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-kube-api-access-mcgl4\") pod \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " Oct 31 14:59:27.993817 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.993779 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-openshift-ca\") pod \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " Oct 31 14:59:27.994062 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.993945 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-xks-ca\") pod \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\" (UID: \"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd\") " Oct 31 14:59:27.994242 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.994205 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-openshift-ca" pod "2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" (UID: "2cd6712d-b0cf-48fc-9fb2-c226efbf92bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:59:27.994385 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.994336 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:27.996145 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.996120 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-xks-ca" pod "2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" (UID: "2cd6712d-b0cf-48fc-9fb2-c226efbf92bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:59:27.996145 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:27.996133 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-kube-api-access-mcgl4" pod "2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" (UID: "2cd6712d-b0cf-48fc-9fb2-c226efbf92bd"). InnerVolumeSpecName "kube-api-access-mcgl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:59:28.095617 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:28.095588 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcgl4\" (UniqueName: \"kubernetes.io/projected/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-kube-api-access-mcgl4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:28.095617 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:28.095615 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:28.717212 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:28.717183 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" Oct 31 14:59:28.717387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:28.717224 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444" event={"ID":"2cd6712d-b0cf-48fc-9fb2-c226efbf92bd","Type":"ContainerDied","Data":"30900af2b3f900b6cbcf53c733954476934e728c04d4a70103d107574559c18f"} Oct 31 14:59:28.717387 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:28.717267 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30900af2b3f900b6cbcf53c733954476934e728c04d4a70103d107574559c18f" Oct 31 14:59:30.317520 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:30.317488 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:35.117626 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:35.117592 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:36.808260 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:36.808223 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:39.899647 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:39.899614 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:40.108045 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:40.108016 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln"] Oct 31 14:59:40.108274 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:40.108251 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" containerID="cri-o://558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6" gracePeriod=30 Oct 31 14:59:40.159846 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:40.159762 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx"] Oct 31 14:59:40.160019 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:40.159979 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" containerID="cri-o://e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1" gracePeriod=30 Oct 31 14:59:40.606007 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:40.605972 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-rj2ln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:59:40.606168 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:40.606032 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:59:42.271273 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.271237 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76"] Oct 31 14:59:42.271692 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.271678 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" containerName="revoke-keys" Oct 31 14:59:42.271736 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.271692 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" containerName="revoke-keys" Oct 31 14:59:42.271795 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.271786 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" containerName="revoke-keys" Oct 31 14:59:42.274947 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.274929 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.330897 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.330864 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.331023 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.330945 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.331023 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.330980 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpd4c\" (UniqueName: \"kubernetes.io/projected/95240a0b-f005-4fe3-9bde-e900244db43a-kube-api-access-gpd4c\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.331701 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.331683 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76"] Oct 31 14:59:42.380229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.380202 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:59:42.380229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.380210 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:59:42.380229 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.380231 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:59:42.380447 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.380277 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 14:59:42.431589 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.431561 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.431778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.431617 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.431778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.431641 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpd4c\" (UniqueName: \"kubernetes.io/projected/95240a0b-f005-4fe3-9bde-e900244db43a-kube-api-access-gpd4c\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.432280 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.432261 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.434137 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.434116 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.490067 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.490039 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpd4c\" (UniqueName: \"kubernetes.io/projected/95240a0b-f005-4fe3-9bde-e900244db43a-kube-api-access-gpd4c\") pod \"maas-api-revoke-sub-e5e746c2-pdk76\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.587935 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.587874 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:42.959051 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:42.959026 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76"] Oct 31 14:59:42.960546 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:59:42.960520 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95240a0b_f005_4fe3_9bde_e900244db43a.slice/crio-95fd4f55314f4a24a3e3dc5782b8ca6810594959ae5273bb8ca6924168e7c30a WatchSource:0}: Error finding container 95fd4f55314f4a24a3e3dc5782b8ca6810594959ae5273bb8ca6924168e7c30a: Status 404 returned error can't find the container with id 95fd4f55314f4a24a3e3dc5782b8ca6810594959ae5273bb8ca6924168e7c30a Oct 31 14:59:43.801778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:43.801740 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" event={"ID":"95240a0b-f005-4fe3-9bde-e900244db43a","Type":"ContainerStarted","Data":"d6065373051993c58a62bc1d4acccc09598b4b12e80ffb312ceab374c62ce4aa"} Oct 31 14:59:43.802118 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:43.801816 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" event={"ID":"95240a0b-f005-4fe3-9bde-e900244db43a","Type":"ContainerStarted","Data":"95fd4f55314f4a24a3e3dc5782b8ca6810594959ae5273bb8ca6924168e7c30a"} Oct 31 14:59:43.917869 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:43.917825 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" podStartSLOduration=1.917810971 podStartE2EDuration="1.917810971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:43.917077619 +0000 UTC m=+829.536629032" watchObservedRunningTime="2026-10-31 14:59:43.917810971 +0000 UTC m=+829.537362381" Oct 31 14:59:44.606801 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:44.606763 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:44.808154 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:44.808118 4865 generic.go:356] "Generic (PLEG): container finished" podID="95240a0b-f005-4fe3-9bde-e900244db43a" containerID="d6065373051993c58a62bc1d4acccc09598b4b12e80ffb312ceab374c62ce4aa" exitCode=0 Oct 31 14:59:44.808547 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:44.808211 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" event={"ID":"95240a0b-f005-4fe3-9bde-e900244db43a","Type":"ContainerDied","Data":"d6065373051993c58a62bc1d4acccc09598b4b12e80ffb312ceab374c62ce4aa"} Oct 31 14:59:45.037966 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.037936 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 14:59:45.445656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.445636 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:59:45.452049 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.452026 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:59:45.456190 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456168 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d62163c-a533-4dfc-bb1e-533085ff1038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d62163c-a533-4dfc-bb1e-533085ff1038-istiod-ca-cert\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456268 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456235 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-kube-api-access-mz4j7\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-kube-api-access-mz4j7\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456319 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456271 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-token\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-token\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456319 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456289 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-data\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456423 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456323 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-credential-socket\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456423 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456371 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-envoy\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456423 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456419 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-kube-api-access-zc8g9\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-kube-api-access-zc8g9\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456448 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-data\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456532 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-podinfo\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456562 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d62163c-a533-4dfc-bb1e-533085ff1038-istiod-ca-cert" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:59:45.456662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456589 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-socket\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456622 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-envoy\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456662 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456654 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-certs\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456689 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-socket\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456720 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-certs\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456765 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istiod-ca-cert\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456809 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-podinfo\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.456998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456843 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-credential-socket\") pod \"2d62163c-a533-4dfc-bb1e-533085ff1038\" (UID: \"2d62163c-a533-4dfc-bb1e-533085ff1038\") " Oct 31 14:59:45.456998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456853 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-data" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.456998 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.456880 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-token\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-token\") pod \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\" (UID: \"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b\") " Oct 31 14:59:45.457360 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457057 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-socket" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.457360 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457185 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-data" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.457487 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457382 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-credential-socket" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.457549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457511 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d62163c-a533-4dfc-bb1e-533085ff1038-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.457549 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457532 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.457656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457547 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.457656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457561 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.457656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457575 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.458178 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457885 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-certs" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.458178 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.457910 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istiod-ca-cert" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:59:45.458178 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.458148 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-certs" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.458393 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.458325 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-socket" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.459370 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.459343 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-credential-socket" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.459546 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.459378 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-token" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:59:45.459660 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.459591 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-envoy" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.459779 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.459668 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-podinfo" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:59:45.459953 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.459925 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-kube-api-access-mz4j7" pod "2d62163c-a533-4dfc-bb1e-533085ff1038" (UID: "2d62163c-a533-4dfc-bb1e-533085ff1038"). InnerVolumeSpecName "kube-api-access-mz4j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:59:45.460301 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.460277 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-token" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:59:45.460979 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.460954 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-envoy" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:59:45.461279 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.461257 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-kube-api-access-zc8g9" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "kube-api-access-zc8g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:59:45.461433 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.461419 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-podinfo" pod "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" (UID: "6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:59:45.558719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558685 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8g9\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-kube-api-access-zc8g9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558713 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558719 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558726 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558738 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558750 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558764 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558775 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558787 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558799 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558810 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558821 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4j7\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-kube-api-access-mz4j7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558834 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.558975 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.558845 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d62163c-a533-4dfc-bb1e-533085ff1038-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:45.814816 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.814783 4865 generic.go:356] "Generic (PLEG): container finished" podID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerID="558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6" exitCode=0 Oct 31 14:59:45.815298 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.814826 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" Oct 31 14:59:45.815298 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.814872 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" event={"ID":"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b","Type":"ContainerDied","Data":"558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6"} Oct 31 14:59:45.815298 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.814928 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln" event={"ID":"6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b","Type":"ContainerDied","Data":"74abe1fa8cf63754ed387feeeacd06c6dcb2720f2f7426b09f05b41e7a141aaf"} Oct 31 14:59:45.815298 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.814961 4865 scope.go:122] "RemoveContainer" containerID="558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6" Oct 31 14:59:45.816423 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.816403 4865 generic.go:356] "Generic (PLEG): container finished" podID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerID="e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1" exitCode=0 Oct 31 14:59:45.816729 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.816453 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" Oct 31 14:59:45.816729 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.816514 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" event={"ID":"2d62163c-a533-4dfc-bb1e-533085ff1038","Type":"ContainerDied","Data":"e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1"} Oct 31 14:59:45.816729 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.816562 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx" event={"ID":"2d62163c-a533-4dfc-bb1e-533085ff1038","Type":"ContainerDied","Data":"9de3fec3727e78901b9a9814a3ed90a96b348c6262d8a877fa2f6e2478b97c1b"} Oct 31 14:59:45.831266 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.831242 4865 scope.go:122] "RemoveContainer" containerID="558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6" Oct 31 14:59:45.831633 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:59:45.831605 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6\": container with ID starting with 558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6 not found: ID does not exist" containerID="558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6" Oct 31 14:59:45.831747 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.831645 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6"} err="failed to get container status \"558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6\": rpc error: code = NotFound desc = could not find container \"558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6\": container with ID starting with 558eaff7ac03cb35f9b7920d0fda7dae2354d5f229fd189a820d17fe898d2bc6 not found: ID does not exist" Oct 31 14:59:45.831747 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.831669 4865 scope.go:122] "RemoveContainer" containerID="e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1" Oct 31 14:59:45.841599 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.841578 4865 scope.go:122] "RemoveContainer" containerID="e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1" Oct 31 14:59:45.841957 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:59:45.841938 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1\": container with ID starting with e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1 not found: ID does not exist" containerID="e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1" Oct 31 14:59:45.842018 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.841967 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1"} err="failed to get container status \"e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1\": rpc error: code = NotFound desc = could not find container \"e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1\": container with ID starting with e3339cf1a300ccde2305b9c3f65efe14569735bc87e65feb728786a9698e42c1 not found: ID does not exist" Oct 31 14:59:45.947209 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.947188 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:45.962161 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.962138 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95240a0b-f005-4fe3-9bde-e900244db43a-kube-api-access-gpd4c\" (UniqueName: \"kubernetes.io/projected/95240a0b-f005-4fe3-9bde-e900244db43a-kube-api-access-gpd4c\") pod \"95240a0b-f005-4fe3-9bde-e900244db43a\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " Oct 31 14:59:45.962290 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.962185 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-xks-ca\") pod \"95240a0b-f005-4fe3-9bde-e900244db43a\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " Oct 31 14:59:45.962290 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.962219 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-openshift-ca\") pod \"95240a0b-f005-4fe3-9bde-e900244db43a\" (UID: \"95240a0b-f005-4fe3-9bde-e900244db43a\") " Oct 31 14:59:45.962579 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.962557 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-openshift-ca" pod "95240a0b-f005-4fe3-9bde-e900244db43a" (UID: "95240a0b-f005-4fe3-9bde-e900244db43a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:59:45.964554 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.964523 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-xks-ca" pod "95240a0b-f005-4fe3-9bde-e900244db43a" (UID: "95240a0b-f005-4fe3-9bde-e900244db43a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:59:45.964664 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.964552 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95240a0b-f005-4fe3-9bde-e900244db43a-kube-api-access-gpd4c" pod "95240a0b-f005-4fe3-9bde-e900244db43a" (UID: "95240a0b-f005-4fe3-9bde-e900244db43a"). InnerVolumeSpecName "kube-api-access-gpd4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:59:45.982638 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.982615 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx"] Oct 31 14:59:45.988374 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:45.988354 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sjwtx"] Oct 31 14:59:46.063631 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.063604 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpd4c\" (UniqueName: \"kubernetes.io/projected/95240a0b-f005-4fe3-9bde-e900244db43a-kube-api-access-gpd4c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:46.063747 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.063635 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:46.063747 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.063646 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95240a0b-f005-4fe3-9bde-e900244db43a-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:59:46.101732 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.101672 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln"] Oct 31 14:59:46.105206 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.105182 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-rj2ln"] Oct 31 14:59:46.495129 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.495101 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" path="/var/lib/kubelet/pods/2d62163c-a533-4dfc-bb1e-533085ff1038/volumes" Oct 31 14:59:46.495558 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.495544 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" path="/var/lib/kubelet/pods/6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b/volumes" Oct 31 14:59:46.825050 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.824973 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" event={"ID":"95240a0b-f005-4fe3-9bde-e900244db43a","Type":"ContainerDied","Data":"95fd4f55314f4a24a3e3dc5782b8ca6810594959ae5273bb8ca6924168e7c30a"} Oct 31 14:59:46.825050 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.825021 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95fd4f55314f4a24a3e3dc5782b8ca6810594959ae5273bb8ca6924168e7c30a" Oct 31 14:59:46.825050 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:46.824984 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76" Oct 31 14:59:47.489621 ip-10-0-131-60 kubenswrapper[4865]: E1031 14:59:47.489584 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 14:59:48.987847 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:48.987814 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:49.399045 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:49.398959 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:50.020744 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:50.020713 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:53.446409 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446372 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq"] Oct 31 14:59:53.446817 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446804 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" Oct 31 14:59:53.446817 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446815 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" Oct 31 14:59:53.446886 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446826 4865 cpu_manager.go:414] "containerMap: removing container" podUID="95240a0b-f005-4fe3-9bde-e900244db43a" containerName="revoke-keys" Oct 31 14:59:53.446886 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446831 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95240a0b-f005-4fe3-9bde-e900244db43a" containerName="revoke-keys" Oct 31 14:59:53.446886 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446857 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" Oct 31 14:59:53.446886 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446862 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" Oct 31 14:59:53.447123 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446931 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2c6faf-2b9a-4ee8-a34f-5118f1ba872b" containerName="istio-proxy" Oct 31 14:59:53.447123 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446939 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="95240a0b-f005-4fe3-9bde-e900244db43a" containerName="revoke-keys" Oct 31 14:59:53.447123 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.446950 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d62163c-a533-4dfc-bb1e-533085ff1038" containerName="istio-proxy" Oct 31 14:59:53.451014 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.450995 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.507215 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.507180 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq"] Oct 31 14:59:53.537068 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.537036 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-dshm\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.537262 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.537234 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-model-cache\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.537340 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.537290 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tmp-dir\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.537457 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.537441 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tls-certs\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.537568 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.537491 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzbk6\" (UniqueName: \"kubernetes.io/projected/511b6f1e-2b64-4b95-8d55-6f7523dcd763-kube-api-access-tzbk6\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.537637 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.537584 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-home\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.555042 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.555020 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"premium-w1-fb2348-kserve-self-signed-certs\"" Oct 31 14:59:53.559764 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.559745 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:53.638851 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.638821 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tls-certs\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639046 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.638861 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzbk6\" (UniqueName: \"kubernetes.io/projected/511b6f1e-2b64-4b95-8d55-6f7523dcd763-kube-api-access-tzbk6\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639046 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.638887 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-home\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639046 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.639013 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-dshm\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639213 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.639153 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-model-cache\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639213 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.639175 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tmp-dir\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639311 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.639286 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-home\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639524 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.639495 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-model-cache\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.639524 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.639509 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tmp-dir\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.641404 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.641383 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-dshm\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.641743 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.641723 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tls-certs\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.696924 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.696845 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzbk6\" (UniqueName: \"kubernetes.io/projected/511b6f1e-2b64-4b95-8d55-6f7523dcd763-kube-api-access-tzbk6\") pod \"premium-w1-fb2348-kserve-9bd649868-jphtq\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:53.762923 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:53.762889 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:54.134302 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:54.134271 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq"] Oct 31 14:59:54.135451 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:59:54.135427 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511b6f1e_2b64_4b95_8d55_6f7523dcd763.slice/crio-9919d43ca5e30ffce0911905f83a7b0e661fcec78e8531212976bd5e182dd503 WatchSource:0}: Error finding container 9919d43ca5e30ffce0911905f83a7b0e661fcec78e8531212976bd5e182dd503: Status 404 returned error can't find the container with id 9919d43ca5e30ffce0911905f83a7b0e661fcec78e8531212976bd5e182dd503 Oct 31 14:59:54.222189 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:54.222152 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:54.899097 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:54.899065 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" event={"ID":"511b6f1e-2b64-4b95-8d55-6f7523dcd763","Type":"ContainerStarted","Data":"17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e"} Oct 31 14:59:54.899519 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:54.899105 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" event={"ID":"511b6f1e-2b64-4b95-8d55-6f7523dcd763","Type":"ContainerStarted","Data":"9919d43ca5e30ffce0911905f83a7b0e661fcec78e8531212976bd5e182dd503"} Oct 31 14:59:55.016036 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.015970 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podStartSLOduration=2.015954096 podStartE2EDuration="2.015954096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:55.014744818 +0000 UTC m=+840.634296230" watchObservedRunningTime="2026-10-31 14:59:55.015954096 +0000 UTC m=+840.635505509" Oct 31 14:59:55.252523 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.252488 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj"] Oct 31 14:59:55.257548 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.257523 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.313514 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.313402 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-190693-openshift-default-dockercfg-qdhg8\"" Oct 31 14:59:55.315907 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.315879 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj"] Oct 31 14:59:55.381651 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.381611 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.381651 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.381658 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6608d87-3bf4-433e-a0b1-04fc46f49384-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.381872 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.381743 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjc6p\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-kube-api-access-qjc6p\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.381872 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.381787 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.381872 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.381809 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.381985 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.381917 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.381985 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.381946 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.382077 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.382010 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.382129 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.382097 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.482935 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.482896 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjc6p\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-kube-api-access-qjc6p\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.482946 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.482966 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.482997 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483014 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483062 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483108 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483081 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483508 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483134 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483508 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483165 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6608d87-3bf4-433e-a0b1-04fc46f49384-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483508 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483407 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483508 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483419 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483519 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483656 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483553 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.483907 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.483883 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6608d87-3bf4-433e-a0b1-04fc46f49384-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.485603 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.485581 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.485692 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.485675 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.541043 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.541009 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.541171 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.541085 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjc6p\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-kube-api-access-qjc6p\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-m2bwj\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.570691 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.570638 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:55.763621 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.763589 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 14:59:55.997864 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:55.997833 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj"] Oct 31 14:59:56.033385 ip-10-0-131-60 kubenswrapper[4865]: W1031 14:59:56.033349 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6608d87_3bf4_433e_a0b1_04fc46f49384.slice/crio-0f02fabfe12aa1c77a9a787d4f3d5a878579f75e5cbda01c010a22eea431dba0 WatchSource:0}: Error finding container 0f02fabfe12aa1c77a9a787d4f3d5a878579f75e5cbda01c010a22eea431dba0: Status 404 returned error can't find the container with id 0f02fabfe12aa1c77a9a787d4f3d5a878579f75e5cbda01c010a22eea431dba0 Oct 31 14:59:56.035836 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:56.035801 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:59:56.035943 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:56.035879 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:59:56.035943 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:56.035921 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 14:59:56.924741 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:56.924708 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" event={"ID":"d6608d87-3bf4-433e-a0b1-04fc46f49384","Type":"ContainerStarted","Data":"f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed"} Oct 31 14:59:56.924951 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:56.924760 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" event={"ID":"d6608d87-3bf4-433e-a0b1-04fc46f49384","Type":"ContainerStarted","Data":"0f02fabfe12aa1c77a9a787d4f3d5a878579f75e5cbda01c010a22eea431dba0"} Oct 31 14:59:57.571285 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:57.571255 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 14:59:57.572778 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:57.572748 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 14:59:57.572915 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:57.572795 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 14:59:58.571825 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:58.571791 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 14:59:58.572209 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:58.571843 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 14:59:59.061725 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:59.061695 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 14:59:59.571755 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:59.571711 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 14:59:59.571947 ip-10-0-131-60 kubenswrapper[4865]: I1031 14:59:59.571791 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 15:00:00.162166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.162108 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podStartSLOduration=5.162092222 podStartE2EDuration="5.162092222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:57.164921925 +0000 UTC m=+842.784473348" watchObservedRunningTime="2026-10-31 15:00:00.162092222 +0000 UTC m=+845.781643642" Oct 31 15:00:00.163969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.163944 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27"] Oct 31 15:00:00.168121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.168102 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7"] Oct 31 15:00:00.168252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.168233 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.172098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.172081 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z"] Oct 31 15:00:00.172228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.172210 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.175344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.175327 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.220535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.220454 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr"] Oct 31 15:00:00.222004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.221980 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:00:00.222314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.222296 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:00:00.222639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.222602 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:00:00.225901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.225882 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.250870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.250847 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.250961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.250934 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686sc\" (UniqueName: \"kubernetes.io/projected/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-kube-api-access-686sc\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.251009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.250999 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfs4\" (UniqueName: \"kubernetes.io/projected/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-kube-api-access-5xfs4\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.251114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.251096 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzlf5\" (UniqueName: \"kubernetes.io/projected/3cfe8640-225e-4bb2-81da-d69f848cd178-kube-api-access-rzlf5\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.251147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.251124 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe8640-225e-4bb2-81da-d69f848cd178-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.251147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.251143 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.276855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.276543 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27"] Oct 31 15:00:00.279878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.279851 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z"] Oct 31 15:00:00.283329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.283303 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7"] Oct 31 15:00:00.286366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.286333 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr"] Oct 31 15:00:00.352630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.352595 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhx26\" (UniqueName: \"kubernetes.io/projected/a4612b02-49ba-41db-95d1-11a21480e55d-kube-api-access-xhx26\") pod \"maas-api-key-cleanup-29890980-4czwr\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.352781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.352701 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4612b02-49ba-41db-95d1-11a21480e55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890980-4czwr\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.352781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.352748 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.352897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.352853 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686sc\" (UniqueName: \"kubernetes.io/projected/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-kube-api-access-686sc\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.352969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.352904 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfs4\" (UniqueName: \"kubernetes.io/projected/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-kube-api-access-5xfs4\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.353036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.352975 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlf5\" (UniqueName: \"kubernetes.io/projected/3cfe8640-225e-4bb2-81da-d69f848cd178-kube-api-access-rzlf5\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.353036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.352994 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe8640-225e-4bb2-81da-d69f848cd178-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.353036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.353009 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.353463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.353444 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.353696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.353679 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.353757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.353704 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe8640-225e-4bb2-81da-d69f848cd178-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.422435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.422350 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlf5\" (UniqueName: \"kubernetes.io/projected/3cfe8640-225e-4bb2-81da-d69f848cd178-kube-api-access-rzlf5\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.422764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.422745 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686sc\" (UniqueName: \"kubernetes.io/projected/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-kube-api-access-686sc\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.423587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.423568 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfs4\" (UniqueName: \"kubernetes.io/projected/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-kube-api-access-5xfs4\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.454461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.454439 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhx26\" (UniqueName: \"kubernetes.io/projected/a4612b02-49ba-41db-95d1-11a21480e55d-kube-api-access-xhx26\") pod \"maas-api-key-cleanup-29890980-4czwr\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.454611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.454514 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4612b02-49ba-41db-95d1-11a21480e55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890980-4czwr\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.455066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.455047 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4612b02-49ba-41db-95d1-11a21480e55d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890980-4czwr\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.528311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.528275 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:00.528785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.528762 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhx26\" (UniqueName: \"kubernetes.io/projected/a4612b02-49ba-41db-95d1-11a21480e55d-kube-api-access-xhx26\") pod \"maas-api-key-cleanup-29890980-4czwr\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.538675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.538655 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:00.548576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.548556 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:00.553498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.553451 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:00.571260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.571234 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:00:00.571399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.571285 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 15:00:00.951710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:00.951681 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27"] Oct 31 15:00:00.952368 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:00.952340 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8029e38f_28ea_4f72_8624_68bd5bbe8ac2.slice/crio-ea161c45a4e8d8fe8235449d90b162f56d5e5bc7e1930847ef7f02bd9d724d99 WatchSource:0}: Error finding container ea161c45a4e8d8fe8235449d90b162f56d5e5bc7e1930847ef7f02bd9d724d99: Status 404 returned error can't find the container with id ea161c45a4e8d8fe8235449d90b162f56d5e5bc7e1930847ef7f02bd9d724d99 Oct 31 15:00:01.030558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.030531 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7"] Oct 31 15:00:01.030655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.030603 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z"] Oct 31 15:00:01.031149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.031133 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr"] Oct 31 15:00:01.136722 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:01.135657 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4612b02_49ba_41db_95d1_11a21480e55d.slice/crio-17220c1b5d30926995bf78c5a3c28b056d22ff0fff0c73553910b56e2461e50a WatchSource:0}: Error finding container 17220c1b5d30926995bf78c5a3c28b056d22ff0fff0c73553910b56e2461e50a: Status 404 returned error can't find the container with id 17220c1b5d30926995bf78c5a3c28b056d22ff0fff0c73553910b56e2461e50a Oct 31 15:00:01.571906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.571878 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:00:01.572053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.571928 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 15:00:01.954735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.954693 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerStarted","Data":"df323ea0cff82ffd3281af5f00bd63d4abb9e34cc2ab6af82c362001ee047a29"} Oct 31 15:00:01.955192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.954748 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerStarted","Data":"17220c1b5d30926995bf78c5a3c28b056d22ff0fff0c73553910b56e2461e50a"} Oct 31 15:00:01.956316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.956293 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerStarted","Data":"f71ef126efcf6d53c866122d04522a86f7d13c0fd6103cd56abec6ceb67376d5"} Oct 31 15:00:01.956457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.956339 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerStarted","Data":"3d66d371b9e63704d0611da32a4f83a46b8051465528468e19a4293237490b25"} Oct 31 15:00:01.957971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.957949 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerStarted","Data":"dc4e46a47a849824e3f1bdacf3f7cf95beb2188992e8fe0740c6c1bec8179310"} Oct 31 15:00:01.958081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.957990 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerStarted","Data":"66e24c0c15eafbe38b1ce12f0e43d0b96a41980977bef819cde4d2c3d57fd15d"} Oct 31 15:00:01.959539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.959518 4865 generic.go:356] "Generic (PLEG): container finished" podID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerID="6e455b0156fbbcf0a2dd625201ecade7580cc921afd8e3dc4341bcd38871d750" exitCode=60 Oct 31 15:00:01.959633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.959553 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" event={"ID":"8029e38f-28ea-4f72-8624-68bd5bbe8ac2","Type":"ContainerDied","Data":"6e455b0156fbbcf0a2dd625201ecade7580cc921afd8e3dc4341bcd38871d750"} Oct 31 15:00:01.959633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.959579 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" event={"ID":"8029e38f-28ea-4f72-8624-68bd5bbe8ac2","Type":"ContainerStarted","Data":"ea161c45a4e8d8fe8235449d90b162f56d5e5bc7e1930847ef7f02bd9d724d99"} Oct 31 15:00:01.959756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.959741 4865 scope.go:122] "RemoveContainer" containerID="6e455b0156fbbcf0a2dd625201ecade7580cc921afd8e3dc4341bcd38871d750" Oct 31 15:00:01.986038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.986015 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl"] Oct 31 15:00:01.990772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:01.990756 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.045127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.045098 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:00:02.046509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.046434 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl"] Oct 31 15:00:02.073595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.073537 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" podStartSLOduration=2.073523468 podStartE2EDuration="2.073523468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:02.070397698 +0000 UTC m=+847.689949110" watchObservedRunningTime="2026-10-31 15:00:02.073523468 +0000 UTC m=+847.693074878" Oct 31 15:00:02.172551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.172464 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.172657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.172576 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnkj\" (UniqueName: \"kubernetes.io/projected/f6260853-430c-4504-a744-5ac2f7a36430-kube-api-access-gsnkj\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.172694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.172663 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.186314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.186263 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" podStartSLOduration=2.18624842 podStartE2EDuration="2.18624842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:02.185458464 +0000 UTC m=+847.805009897" watchObservedRunningTime="2026-10-31 15:00:02.18624842 +0000 UTC m=+847.805799830" Oct 31 15:00:02.273707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.273678 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.273821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.273774 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.273858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.273826 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnkj\" (UniqueName: \"kubernetes.io/projected/f6260853-430c-4504-a744-5ac2f7a36430-kube-api-access-gsnkj\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.274342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.274321 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.276147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.276126 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.333223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.333191 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnkj\" (UniqueName: \"kubernetes.io/projected/f6260853-430c-4504-a744-5ac2f7a36430-kube-api-access-gsnkj\") pod \"maas-api-revoke-sub-808a6b23-hxkwl\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.530541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.530494 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" podStartSLOduration=2.530462346 podStartE2EDuration="2.530462346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:02.53019266 +0000 UTC m=+848.149744072" watchObservedRunningTime="2026-10-31 15:00:02.530462346 +0000 UTC m=+848.150013808" Oct 31 15:00:02.571879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.571850 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:00:02.572016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.571901 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 15:00:02.618766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.618742 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:02.934301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.934264 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6"] Oct 31 15:00:02.939077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.939051 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:02.967980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.967949 4865 generic.go:356] "Generic (PLEG): container finished" podID="a4612b02-49ba-41db-95d1-11a21480e55d" containerID="df323ea0cff82ffd3281af5f00bd63d4abb9e34cc2ab6af82c362001ee047a29" exitCode=60 Oct 31 15:00:02.968741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.968059 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerDied","Data":"df323ea0cff82ffd3281af5f00bd63d4abb9e34cc2ab6af82c362001ee047a29"} Oct 31 15:00:02.968741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.968281 4865 scope.go:122] "RemoveContainer" containerID="df323ea0cff82ffd3281af5f00bd63d4abb9e34cc2ab6af82c362001ee047a29" Oct 31 15:00:02.970109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.970038 4865 generic.go:356] "Generic (PLEG): container finished" podID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerID="f71ef126efcf6d53c866122d04522a86f7d13c0fd6103cd56abec6ceb67376d5" exitCode=60 Oct 31 15:00:02.970435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.970099 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerDied","Data":"f71ef126efcf6d53c866122d04522a86f7d13c0fd6103cd56abec6ceb67376d5"} Oct 31 15:00:02.970534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.970319 4865 scope.go:122] "RemoveContainer" containerID="f71ef126efcf6d53c866122d04522a86f7d13c0fd6103cd56abec6ceb67376d5" Oct 31 15:00:02.971979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.971962 4865 generic.go:356] "Generic (PLEG): container finished" podID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerID="dc4e46a47a849824e3f1bdacf3f7cf95beb2188992e8fe0740c6c1bec8179310" exitCode=60 Oct 31 15:00:02.972061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.972047 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerDied","Data":"dc4e46a47a849824e3f1bdacf3f7cf95beb2188992e8fe0740c6c1bec8179310"} Oct 31 15:00:02.972289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.972277 4865 scope.go:122] "RemoveContainer" containerID="dc4e46a47a849824e3f1bdacf3f7cf95beb2188992e8fe0740c6c1bec8179310" Oct 31 15:00:02.974743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.974727 4865 generic.go:356] "Generic (PLEG): container finished" podID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerID="3199b1e0e6296acbf113ac6720fc1ce286ab7b76074f95061325632c4c1a5846" exitCode=60 Oct 31 15:00:02.974822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.974803 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" event={"ID":"8029e38f-28ea-4f72-8624-68bd5bbe8ac2","Type":"ContainerDied","Data":"3199b1e0e6296acbf113ac6720fc1ce286ab7b76074f95061325632c4c1a5846"} Oct 31 15:00:02.974861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.974832 4865 scope.go:122] "RemoveContainer" containerID="6e455b0156fbbcf0a2dd625201ecade7580cc921afd8e3dc4341bcd38871d750" Oct 31 15:00:02.975089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:02.975077 4865 scope.go:122] "RemoveContainer" containerID="3199b1e0e6296acbf113ac6720fc1ce286ab7b76074f95061325632c4c1a5846" Oct 31 15:00:02.975239 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:02.975223 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27_odh-ai-gateway-infra(8029e38f-28ea-4f72-8624-68bd5bbe8ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" Oct 31 15:00:03.002637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.002605 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6"] Oct 31 15:00:03.030522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.030493 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-029d26-openshift-default-dockercfg-lxm75\"" Oct 31 15:00:03.049204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.047974 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl"] Oct 31 15:00:03.051002 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:03.050972 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6260853_430c_4504_a744_5ac2f7a36430.slice/crio-b45b50e0873d3b3c8b988fac1c494c0dd7b3292c9ffc6b8f50c589a48b91a560 WatchSource:0}: Error finding container b45b50e0873d3b3c8b988fac1c494c0dd7b3292c9ffc6b8f50c589a48b91a560: Status 404 returned error can't find the container with id b45b50e0873d3b3c8b988fac1c494c0dd7b3292c9ffc6b8f50c589a48b91a560 Oct 31 15:00:03.083481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.083434 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.083582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.083505 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.084322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.084299 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.084429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.084373 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.084429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.084392 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.084429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.084413 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a516a3-4e43-461a-adef-41d03726eec5-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.084615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.084452 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a516a3-4e43-461a-adef-41d03726eec5-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.084615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.084487 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.084615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.084515 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwqw\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-kube-api-access-bvwqw\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.185905 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.185952 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.185990 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186016 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186031 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a516a3-4e43-461a-adef-41d03726eec5-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186257 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a516a3-4e43-461a-adef-41d03726eec5-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186279 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186301 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwqw\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-kube-api-access-bvwqw\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186439 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186509 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186547 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.186872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.186849 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.187040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.187020 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a516a3-4e43-461a-adef-41d03726eec5-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.195104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.195078 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.195394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.195376 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a516a3-4e43-461a-adef-41d03726eec5-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.243912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.243886 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.244007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.243946 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwqw\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-kube-api-access-bvwqw\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.258674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.258656 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:03.576102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.576074 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 15:00:03.576500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.576462 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 15:00:03.577106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.577087 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 15:00:03.642041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.642014 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6"] Oct 31 15:00:03.643293 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:03.643272 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a516a3_4e43_461a_adef_41d03726eec5.slice/crio-8a48b77036a1607f46d738b64a20586fbc00cc48a02c5835eae8bbd8497c75e0 WatchSource:0}: Error finding container 8a48b77036a1607f46d738b64a20586fbc00cc48a02c5835eae8bbd8497c75e0: Status 404 returned error can't find the container with id 8a48b77036a1607f46d738b64a20586fbc00cc48a02c5835eae8bbd8497c75e0 Oct 31 15:00:03.646099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.646065 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:03.646190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.646130 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:03.646190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.646167 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:03.763339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.763262 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:00:03.780929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.780901 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:00:03.921987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:03.921953 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:04.005395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.005374 4865 scope.go:122] "RemoveContainer" containerID="3199b1e0e6296acbf113ac6720fc1ce286ab7b76074f95061325632c4c1a5846" Oct 31 15:00:04.006765 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:04.006197 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27_odh-ai-gateway-infra(8029e38f-28ea-4f72-8624-68bd5bbe8ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" Oct 31 15:00:04.007270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.007225 4865 generic.go:356] "Generic (PLEG): container finished" podID="f6260853-430c-4504-a744-5ac2f7a36430" containerID="8105a5a0480bf72d170dfe6d6472f5172421fb72ce3f7f84bc6153fc43fd68b4" exitCode=0 Oct 31 15:00:04.007411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.007385 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" event={"ID":"f6260853-430c-4504-a744-5ac2f7a36430","Type":"ContainerDied","Data":"8105a5a0480bf72d170dfe6d6472f5172421fb72ce3f7f84bc6153fc43fd68b4"} Oct 31 15:00:04.007580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.007436 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" event={"ID":"f6260853-430c-4504-a744-5ac2f7a36430","Type":"ContainerStarted","Data":"b45b50e0873d3b3c8b988fac1c494c0dd7b3292c9ffc6b8f50c589a48b91a560"} Oct 31 15:00:04.012506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.009707 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" event={"ID":"88a516a3-4e43-461a-adef-41d03726eec5","Type":"ContainerStarted","Data":"0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa"} Oct 31 15:00:04.012506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.009763 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" event={"ID":"88a516a3-4e43-461a-adef-41d03726eec5","Type":"ContainerStarted","Data":"8a48b77036a1607f46d738b64a20586fbc00cc48a02c5835eae8bbd8497c75e0"} Oct 31 15:00:04.012506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.012019 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerStarted","Data":"3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1"} Oct 31 15:00:04.012506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.012280 4865 scope.go:122] "RemoveContainer" containerID="3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1" Oct 31 15:00:04.012750 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:04.012566 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890980-4czwr_odh-ai-gateway-infra(a4612b02-49ba-41db-95d1-11a21480e55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" Oct 31 15:00:04.014306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.014248 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerStarted","Data":"29d25fe850a8e58968876989f3b2388f7f9c946ad4610cd086c0d02f277654d6"} Oct 31 15:00:04.016758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.016733 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerStarted","Data":"0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d"} Oct 31 15:00:04.017018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.017000 4865 scope.go:122] "RemoveContainer" containerID="0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d" Oct 31 15:00:04.017213 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:04.017192 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7_odh-ai-gateway-infra(8a7a9e85-b0f5-4f75-8e17-5e54d10ff936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" Oct 31 15:00:04.034541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.034520 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:00:04.259193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.259161 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:04.260481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.260436 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:00:04.260606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.260543 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 15:00:04.838845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:04.838785 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podStartSLOduration=2.838768037 podStartE2EDuration="2.838768037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:04.716790489 +0000 UTC m=+850.336341901" watchObservedRunningTime="2026-10-31 15:00:04.838768037 +0000 UTC m=+850.458319449" Oct 31 15:00:05.029684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.029645 4865 generic.go:356] "Generic (PLEG): container finished" podID="a4612b02-49ba-41db-95d1-11a21480e55d" containerID="3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1" exitCode=60 Oct 31 15:00:05.030148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.029733 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerDied","Data":"3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1"} Oct 31 15:00:05.030148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.029837 4865 scope.go:122] "RemoveContainer" containerID="df323ea0cff82ffd3281af5f00bd63d4abb9e34cc2ab6af82c362001ee047a29" Oct 31 15:00:05.030148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.029960 4865 scope.go:122] "RemoveContainer" containerID="3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1" Oct 31 15:00:05.030329 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:05.030177 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890980-4czwr_odh-ai-gateway-infra(a4612b02-49ba-41db-95d1-11a21480e55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" Oct 31 15:00:05.032709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.032687 4865 generic.go:356] "Generic (PLEG): container finished" podID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerID="29d25fe850a8e58968876989f3b2388f7f9c946ad4610cd086c0d02f277654d6" exitCode=60 Oct 31 15:00:05.032832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.032774 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerDied","Data":"29d25fe850a8e58968876989f3b2388f7f9c946ad4610cd086c0d02f277654d6"} Oct 31 15:00:05.032966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.032950 4865 scope.go:122] "RemoveContainer" containerID="29d25fe850a8e58968876989f3b2388f7f9c946ad4610cd086c0d02f277654d6" Oct 31 15:00:05.033194 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:05.033173 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z_odh-ai-gateway-infra(3cfe8640-225e-4bb2-81da-d69f848cd178)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" Oct 31 15:00:05.034624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.034602 4865 generic.go:356] "Generic (PLEG): container finished" podID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerID="0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d" exitCode=60 Oct 31 15:00:05.034744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.034673 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerDied","Data":"0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d"} Oct 31 15:00:05.034865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.034847 4865 scope.go:122] "RemoveContainer" containerID="0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d" Oct 31 15:00:05.035078 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:05.035055 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7_odh-ai-gateway-infra(8a7a9e85-b0f5-4f75-8e17-5e54d10ff936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" Oct 31 15:00:05.042954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.042937 4865 scope.go:122] "RemoveContainer" containerID="f71ef126efcf6d53c866122d04522a86f7d13c0fd6103cd56abec6ceb67376d5" Oct 31 15:00:05.057330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.057312 4865 scope.go:122] "RemoveContainer" containerID="dc4e46a47a849824e3f1bdacf3f7cf95beb2188992e8fe0740c6c1bec8179310" Oct 31 15:00:05.165274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.165240 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:00:05.248355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.248330 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:05.259196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.259168 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:00:05.259292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.259219 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 15:00:05.325026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.324996 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6260853-430c-4504-a744-5ac2f7a36430-kube-api-access-gsnkj\" (UniqueName: \"kubernetes.io/projected/f6260853-430c-4504-a744-5ac2f7a36430-kube-api-access-gsnkj\") pod \"f6260853-430c-4504-a744-5ac2f7a36430\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " Oct 31 15:00:05.325214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.325084 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-xks-ca\") pod \"f6260853-430c-4504-a744-5ac2f7a36430\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " Oct 31 15:00:05.325214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.325171 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-openshift-ca\") pod \"f6260853-430c-4504-a744-5ac2f7a36430\" (UID: \"f6260853-430c-4504-a744-5ac2f7a36430\") " Oct 31 15:00:05.326605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.326572 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-openshift-ca" pod "f6260853-430c-4504-a744-5ac2f7a36430" (UID: "f6260853-430c-4504-a744-5ac2f7a36430"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:05.334448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.334051 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6260853-430c-4504-a744-5ac2f7a36430-kube-api-access-gsnkj" pod "f6260853-430c-4504-a744-5ac2f7a36430" (UID: "f6260853-430c-4504-a744-5ac2f7a36430"). InnerVolumeSpecName "kube-api-access-gsnkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:05.334627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.334568 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-xks-ca" pod "f6260853-430c-4504-a744-5ac2f7a36430" (UID: "f6260853-430c-4504-a744-5ac2f7a36430"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:00:05.427419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.427344 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnkj\" (UniqueName: \"kubernetes.io/projected/f6260853-430c-4504-a744-5ac2f7a36430-kube-api-access-gsnkj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:05.427649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.427622 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:05.427732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:05.427723 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6260853-430c-4504-a744-5ac2f7a36430-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:06.065448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.063238 4865 scope.go:122] "RemoveContainer" containerID="3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1" Oct 31 15:00:06.065448 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:06.063663 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890980-4czwr_odh-ai-gateway-infra(a4612b02-49ba-41db-95d1-11a21480e55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" Oct 31 15:00:06.070186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.070156 4865 scope.go:122] "RemoveContainer" containerID="29d25fe850a8e58968876989f3b2388f7f9c946ad4610cd086c0d02f277654d6" Oct 31 15:00:06.070528 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:06.070461 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z_odh-ai-gateway-infra(3cfe8640-225e-4bb2-81da-d69f848cd178)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" Oct 31 15:00:06.075671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.075645 4865 scope.go:122] "RemoveContainer" containerID="0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d" Oct 31 15:00:06.075944 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:06.075917 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7_odh-ai-gateway-infra(8a7a9e85-b0f5-4f75-8e17-5e54d10ff936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" Oct 31 15:00:06.077833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.077809 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" Oct 31 15:00:06.077938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.077854 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl" event={"ID":"f6260853-430c-4504-a744-5ac2f7a36430","Type":"ContainerDied","Data":"b45b50e0873d3b3c8b988fac1c494c0dd7b3292c9ffc6b8f50c589a48b91a560"} Oct 31 15:00:06.077938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.077895 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b45b50e0873d3b3c8b988fac1c494c0dd7b3292c9ffc6b8f50c589a48b91a560" Oct 31 15:00:06.261854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.261813 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:00:06.262121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:06.262092 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 15:00:07.259314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:07.259276 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:00:07.259744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:07.259345 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 15:00:08.259017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:08.258979 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:00:08.259195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:08.259028 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 15:00:08.644455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:08.644368 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:09.259855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:09.259816 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:00:09.260033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:09.259884 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 15:00:10.260032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:10.260003 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:00:10.260414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:10.260055 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 15:00:11.721045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:11.721015 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:11.721566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:11.721442 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:11.722328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:11.722307 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:00:13.398044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:13.398006 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:14.496655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:14.496626 4865 scope.go:122] "RemoveContainer" containerID="0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d" Oct 31 15:00:14.497077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:14.496947 4865 scope.go:122] "RemoveContainer" containerID="3199b1e0e6296acbf113ac6720fc1ce286ab7b76074f95061325632c4c1a5846" Oct 31 15:00:15.153907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:15.153822 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerStarted","Data":"592d31b1bfe9ae3026e9a39a914f4bb87453dfb942a747e10e854f44b13145ee"} Oct 31 15:00:15.155690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:15.155664 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" event={"ID":"8029e38f-28ea-4f72-8624-68bd5bbe8ac2","Type":"ContainerStarted","Data":"0eebd43cd05adeb4dbe9271705916845071d20296aa37ead91576a67c701dac5"} Oct 31 15:00:15.385931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:15.385883 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" podStartSLOduration=15.385869512 podStartE2EDuration="15.385869512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:15.3841992 +0000 UTC m=+861.003750603" watchObservedRunningTime="2026-10-31 15:00:15.385869512 +0000 UTC m=+861.005420923" Oct 31 15:00:15.489595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:15.489573 4865 scope.go:122] "RemoveContainer" containerID="3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1" Oct 31 15:00:16.163940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.163853 4865 generic.go:356] "Generic (PLEG): container finished" podID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerID="0eebd43cd05adeb4dbe9271705916845071d20296aa37ead91576a67c701dac5" exitCode=60 Oct 31 15:00:16.164385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.163932 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" event={"ID":"8029e38f-28ea-4f72-8624-68bd5bbe8ac2","Type":"ContainerDied","Data":"0eebd43cd05adeb4dbe9271705916845071d20296aa37ead91576a67c701dac5"} Oct 31 15:00:16.164385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.164028 4865 scope.go:122] "RemoveContainer" containerID="3199b1e0e6296acbf113ac6720fc1ce286ab7b76074f95061325632c4c1a5846" Oct 31 15:00:16.164385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.164355 4865 scope.go:122] "RemoveContainer" containerID="0eebd43cd05adeb4dbe9271705916845071d20296aa37ead91576a67c701dac5" Oct 31 15:00:16.164665 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:16.164640 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27_odh-ai-gateway-infra(8029e38f-28ea-4f72-8624-68bd5bbe8ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" Oct 31 15:00:16.166230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.166193 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerStarted","Data":"1d64e04d56e8ef1ec4b8d6e13161a03282dc31cc2e9cc8e80de6afd2bd432fe3"} Oct 31 15:00:16.168069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.168050 4865 generic.go:356] "Generic (PLEG): container finished" podID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerID="592d31b1bfe9ae3026e9a39a914f4bb87453dfb942a747e10e854f44b13145ee" exitCode=60 Oct 31 15:00:16.168153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.168085 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerDied","Data":"592d31b1bfe9ae3026e9a39a914f4bb87453dfb942a747e10e854f44b13145ee"} Oct 31 15:00:16.168415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.168332 4865 scope.go:122] "RemoveContainer" containerID="592d31b1bfe9ae3026e9a39a914f4bb87453dfb942a747e10e854f44b13145ee" Oct 31 15:00:16.168658 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:16.168571 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7_odh-ai-gateway-infra(8a7a9e85-b0f5-4f75-8e17-5e54d10ff936)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" Oct 31 15:00:16.175882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.175863 4865 scope.go:122] "RemoveContainer" containerID="0ed037691836ada820aa7023e9bd7e5ef8857cc5ee65495e8be8ea58cc80ab1d" Oct 31 15:00:16.280642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.280612 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7"] Oct 31 15:00:16.394537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:16.394504 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27"] Oct 31 15:00:17.184107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.184072 4865 generic.go:356] "Generic (PLEG): container finished" podID="a4612b02-49ba-41db-95d1-11a21480e55d" containerID="1d64e04d56e8ef1ec4b8d6e13161a03282dc31cc2e9cc8e80de6afd2bd432fe3" exitCode=60 Oct 31 15:00:17.184569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.184136 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerDied","Data":"1d64e04d56e8ef1ec4b8d6e13161a03282dc31cc2e9cc8e80de6afd2bd432fe3"} Oct 31 15:00:17.184569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.184213 4865 scope.go:122] "RemoveContainer" containerID="3f32eec0dfc7cb780878da6c0ec315135a7685c5915ca72f6c1681d1d4c044e1" Oct 31 15:00:17.184569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.184559 4865 scope.go:122] "RemoveContainer" containerID="1d64e04d56e8ef1ec4b8d6e13161a03282dc31cc2e9cc8e80de6afd2bd432fe3" Oct 31 15:00:17.185157 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:17.184845 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890980-4czwr_odh-ai-gateway-infra(a4612b02-49ba-41db-95d1-11a21480e55d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" Oct 31 15:00:17.325661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.325637 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:17.365992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.365967 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:17.405655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.405624 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr"] Oct 31 15:00:17.426166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.426094 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-maas-api-service-ca\") pod \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " Oct 31 15:00:17.426335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.426206 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-kube-api-access-5xfs4\" (UniqueName: \"kubernetes.io/projected/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-kube-api-access-5xfs4\") pod \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " Oct 31 15:00:17.426335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.426267 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-kube-api-access-686sc\" (UniqueName: \"kubernetes.io/projected/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-kube-api-access-686sc\") pod \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\" (UID: \"8029e38f-28ea-4f72-8624-68bd5bbe8ac2\") " Oct 31 15:00:17.426483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.426353 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-maas-api-service-ca\") pod \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\" (UID: \"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936\") " Oct 31 15:00:17.427150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.427119 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-maas-api-service-ca" pod "8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" (UID: "8a7a9e85-b0f5-4f75-8e17-5e54d10ff936"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:17.427256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.427124 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-maas-api-service-ca" pod "8029e38f-28ea-4f72-8624-68bd5bbe8ac2" (UID: "8029e38f-28ea-4f72-8624-68bd5bbe8ac2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:17.428668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.428644 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-kube-api-access-5xfs4" pod "8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" (UID: "8a7a9e85-b0f5-4f75-8e17-5e54d10ff936"). InnerVolumeSpecName "kube-api-access-5xfs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:17.428866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.428843 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-kube-api-access-686sc" pod "8029e38f-28ea-4f72-8624-68bd5bbe8ac2" (UID: "8029e38f-28ea-4f72-8624-68bd5bbe8ac2"). InnerVolumeSpecName "kube-api-access-686sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:17.489268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.489239 4865 scope.go:122] "RemoveContainer" containerID="29d25fe850a8e58968876989f3b2388f7f9c946ad4610cd086c0d02f277654d6" Oct 31 15:00:17.489968 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:17.489919 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:00:17.528278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.528251 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:17.528397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.528279 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:17.528397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.528299 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xfs4\" (UniqueName: \"kubernetes.io/projected/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936-kube-api-access-5xfs4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:17.528397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:17.528314 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686sc\" (UniqueName: \"kubernetes.io/projected/8029e38f-28ea-4f72-8624-68bd5bbe8ac2-kube-api-access-686sc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:18.124377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.124288 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:18.192387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.192349 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerStarted","Data":"0ce9b31ea5005a8c6b16b0d1ec1c8da1d2d8db21353b9f43f42cacc24282bc1f"} Oct 31 15:00:18.193735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.193715 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" Oct 31 15:00:18.193882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.193734 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7" event={"ID":"8a7a9e85-b0f5-4f75-8e17-5e54d10ff936","Type":"ContainerDied","Data":"66e24c0c15eafbe38b1ce12f0e43d0b96a41980977bef819cde4d2c3d57fd15d"} Oct 31 15:00:18.193882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.193786 4865 scope.go:122] "RemoveContainer" containerID="592d31b1bfe9ae3026e9a39a914f4bb87453dfb942a747e10e854f44b13145ee" Oct 31 15:00:18.195259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.195236 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" event={"ID":"8029e38f-28ea-4f72-8624-68bd5bbe8ac2","Type":"ContainerDied","Data":"ea161c45a4e8d8fe8235449d90b162f56d5e5bc7e1930847ef7f02bd9d724d99"} Oct 31 15:00:18.195408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.195385 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27" Oct 31 15:00:18.215966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.215920 4865 scope.go:122] "RemoveContainer" containerID="0eebd43cd05adeb4dbe9271705916845071d20296aa37ead91576a67c701dac5" Oct 31 15:00:18.359636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.359609 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:18.439246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.439217 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4612b02-49ba-41db-95d1-11a21480e55d-kube-api-access-xhx26\" (UniqueName: \"kubernetes.io/projected/a4612b02-49ba-41db-95d1-11a21480e55d-kube-api-access-xhx26\") pod \"a4612b02-49ba-41db-95d1-11a21480e55d\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " Oct 31 15:00:18.439380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.439332 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4612b02-49ba-41db-95d1-11a21480e55d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4612b02-49ba-41db-95d1-11a21480e55d-maas-api-service-ca\") pod \"a4612b02-49ba-41db-95d1-11a21480e55d\" (UID: \"a4612b02-49ba-41db-95d1-11a21480e55d\") " Oct 31 15:00:18.439690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.439666 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4612b02-49ba-41db-95d1-11a21480e55d-maas-api-service-ca" pod "a4612b02-49ba-41db-95d1-11a21480e55d" (UID: "a4612b02-49ba-41db-95d1-11a21480e55d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:18.441461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.441434 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4612b02-49ba-41db-95d1-11a21480e55d-kube-api-access-xhx26" pod "a4612b02-49ba-41db-95d1-11a21480e55d" (UID: "a4612b02-49ba-41db-95d1-11a21480e55d"). InnerVolumeSpecName "kube-api-access-xhx26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:18.538655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.538624 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7"] Oct 31 15:00:18.541062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.541037 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhx26\" (UniqueName: \"kubernetes.io/projected/a4612b02-49ba-41db-95d1-11a21480e55d-kube-api-access-xhx26\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:18.541062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.541063 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4612b02-49ba-41db-95d1-11a21480e55d-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:18.542054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.542027 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890980-25qf7"] Oct 31 15:00:18.652416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.652341 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27"] Oct 31 15:00:18.657150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:18.657122 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890980-mqb27"] Oct 31 15:00:19.208564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.208536 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" Oct 31 15:00:19.208564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.208554 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr" event={"ID":"a4612b02-49ba-41db-95d1-11a21480e55d","Type":"ContainerDied","Data":"17220c1b5d30926995bf78c5a3c28b056d22ff0fff0c73553910b56e2461e50a"} Oct 31 15:00:19.209099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.208614 4865 scope.go:122] "RemoveContainer" containerID="1d64e04d56e8ef1ec4b8d6e13161a03282dc31cc2e9cc8e80de6afd2bd432fe3" Oct 31 15:00:19.210839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.210806 4865 generic.go:356] "Generic (PLEG): container finished" podID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerID="0ce9b31ea5005a8c6b16b0d1ec1c8da1d2d8db21353b9f43f42cacc24282bc1f" exitCode=60 Oct 31 15:00:19.211029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.210912 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerDied","Data":"0ce9b31ea5005a8c6b16b0d1ec1c8da1d2d8db21353b9f43f42cacc24282bc1f"} Oct 31 15:00:19.211346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.211325 4865 scope.go:122] "RemoveContainer" containerID="0ce9b31ea5005a8c6b16b0d1ec1c8da1d2d8db21353b9f43f42cacc24282bc1f" Oct 31 15:00:19.211701 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:19.211665 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z_odh-ai-gateway-infra(3cfe8640-225e-4bb2-81da-d69f848cd178)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" Oct 31 15:00:19.218538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.218520 4865 scope.go:122] "RemoveContainer" containerID="29d25fe850a8e58968876989f3b2388f7f9c946ad4610cd086c0d02f277654d6" Oct 31 15:00:19.319872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.319846 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z"] Oct 31 15:00:19.378059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.378028 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr"] Oct 31 15:00:19.383634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:19.383610 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890980-4czwr"] Oct 31 15:00:20.366979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.366957 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:20.460375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.460341 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfe8640-225e-4bb2-81da-d69f848cd178-kube-api-access-rzlf5\" (UniqueName: \"kubernetes.io/projected/3cfe8640-225e-4bb2-81da-d69f848cd178-kube-api-access-rzlf5\") pod \"3cfe8640-225e-4bb2-81da-d69f848cd178\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " Oct 31 15:00:20.460559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.460518 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfe8640-225e-4bb2-81da-d69f848cd178-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe8640-225e-4bb2-81da-d69f848cd178-maas-api-service-ca\") pod \"3cfe8640-225e-4bb2-81da-d69f848cd178\" (UID: \"3cfe8640-225e-4bb2-81da-d69f848cd178\") " Oct 31 15:00:20.460884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.460854 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfe8640-225e-4bb2-81da-d69f848cd178-maas-api-service-ca" pod "3cfe8640-225e-4bb2-81da-d69f848cd178" (UID: "3cfe8640-225e-4bb2-81da-d69f848cd178"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:20.462534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.462509 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfe8640-225e-4bb2-81da-d69f848cd178-kube-api-access-rzlf5" pod "3cfe8640-225e-4bb2-81da-d69f848cd178" (UID: "3cfe8640-225e-4bb2-81da-d69f848cd178"). InnerVolumeSpecName "kube-api-access-rzlf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:20.497325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.497251 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" path="/var/lib/kubelet/pods/8029e38f-28ea-4f72-8624-68bd5bbe8ac2/volumes" Oct 31 15:00:20.497668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.497653 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" path="/var/lib/kubelet/pods/8a7a9e85-b0f5-4f75-8e17-5e54d10ff936/volumes" Oct 31 15:00:20.497991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.497977 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" path="/var/lib/kubelet/pods/a4612b02-49ba-41db-95d1-11a21480e55d/volumes" Oct 31 15:00:20.562346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.562320 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzlf5\" (UniqueName: \"kubernetes.io/projected/3cfe8640-225e-4bb2-81da-d69f848cd178-kube-api-access-rzlf5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:20.562346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:20.562344 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe8640-225e-4bb2-81da-d69f848cd178-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:21.229595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:21.229570 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" Oct 31 15:00:21.229820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:21.229602 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z" event={"ID":"3cfe8640-225e-4bb2-81da-d69f848cd178","Type":"ContainerDied","Data":"3d66d371b9e63704d0611da32a4f83a46b8051465528468e19a4293237490b25"} Oct 31 15:00:21.229820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:21.229668 4865 scope.go:122] "RemoveContainer" containerID="0ce9b31ea5005a8c6b16b0d1ec1c8da1d2d8db21353b9f43f42cacc24282bc1f" Oct 31 15:00:21.399618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:21.399569 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z"] Oct 31 15:00:21.408078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:21.408029 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890980-z6q4z"] Oct 31 15:00:22.496893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.496859 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" path="/var/lib/kubelet/pods/3cfe8640-225e-4bb2-81da-d69f848cd178/volumes" Oct 31 15:00:22.575099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575067 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc"] Oct 31 15:00:22.575515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575502 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:22.575563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575516 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:22.575563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575529 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575534 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575541 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:22.575563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575546 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:22.575563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575553 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575557 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575576 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575581 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575586 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575590 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575597 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575601 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575615 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575620 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575625 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575629 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575636 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575641 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575649 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f6260853-430c-4504-a744-5ac2f7a36430" containerName="revoke-keys" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575653 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6260853-430c-4504-a744-5ac2f7a36430" containerName="revoke-keys" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575731 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575739 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575745 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575753 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575758 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575763 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4612b02-49ba-41db-95d1-11a21480e55d" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575769 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575774 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575777 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575782 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6260853-430c-4504-a744-5ac2f7a36430" containerName="revoke-keys" Oct 31 15:00:22.575789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.575788 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:22.580272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.580256 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.634898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.634866 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc"] Oct 31 15:00:22.682640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.682602 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:00:22.682819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.682672 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:00:22.682819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.682718 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:00:22.682819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.682735 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:00:22.692176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.692150 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.692299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.692233 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlwd\" (UniqueName: \"kubernetes.io/projected/bd908f19-bab0-4fba-ad9d-060798e799b7-kube-api-access-4mlwd\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.692299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.692284 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.793910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.793831 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlwd\" (UniqueName: \"kubernetes.io/projected/bd908f19-bab0-4fba-ad9d-060798e799b7-kube-api-access-4mlwd\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.794059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.793925 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.794059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.793987 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.794587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.794558 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.796403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.796382 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.851664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.851640 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlwd\" (UniqueName: \"kubernetes.io/projected/bd908f19-bab0-4fba-ad9d-060798e799b7-kube-api-access-4mlwd\") pod \"maas-api-revoke-sub-e2552ae1-pj5dc\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:22.852775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.852751 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:22.893570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:22.893538 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:23.281205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:23.281181 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc"] Oct 31 15:00:23.282084 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:23.282056 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd908f19_bab0_4fba_ad9d_060798e799b7.slice/crio-3031a6b0160739546ffef0867745862b5886662a2fffcbfbf9e1aa725b3d3a97 WatchSource:0}: Error finding container 3031a6b0160739546ffef0867745862b5886662a2fffcbfbf9e1aa725b3d3a97: Status 404 returned error can't find the container with id 3031a6b0160739546ffef0867745862b5886662a2fffcbfbf9e1aa725b3d3a97 Oct 31 15:00:24.264217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:24.264182 4865 generic.go:356] "Generic (PLEG): container finished" podID="bd908f19-bab0-4fba-ad9d-060798e799b7" containerID="5b5d16c6bf048ef41c7a57ee50dafe73686791ea3758df40d7569be1f46327d1" exitCode=0 Oct 31 15:00:24.264629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:24.264265 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" event={"ID":"bd908f19-bab0-4fba-ad9d-060798e799b7","Type":"ContainerDied","Data":"5b5d16c6bf048ef41c7a57ee50dafe73686791ea3758df40d7569be1f46327d1"} Oct 31 15:00:24.264629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:24.264314 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" event={"ID":"bd908f19-bab0-4fba-ad9d-060798e799b7","Type":"ContainerStarted","Data":"3031a6b0160739546ffef0867745862b5886662a2fffcbfbf9e1aa725b3d3a97"} Oct 31 15:00:25.413770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.413744 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:25.522087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.522016 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-openshift-ca\") pod \"bd908f19-bab0-4fba-ad9d-060798e799b7\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " Oct 31 15:00:25.522087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.522069 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd908f19-bab0-4fba-ad9d-060798e799b7-kube-api-access-4mlwd\" (UniqueName: \"kubernetes.io/projected/bd908f19-bab0-4fba-ad9d-060798e799b7-kube-api-access-4mlwd\") pod \"bd908f19-bab0-4fba-ad9d-060798e799b7\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " Oct 31 15:00:25.522336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.522172 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-xks-ca\") pod \"bd908f19-bab0-4fba-ad9d-060798e799b7\" (UID: \"bd908f19-bab0-4fba-ad9d-060798e799b7\") " Oct 31 15:00:25.522398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.522363 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-openshift-ca" pod "bd908f19-bab0-4fba-ad9d-060798e799b7" (UID: "bd908f19-bab0-4fba-ad9d-060798e799b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:25.524429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.524406 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd908f19-bab0-4fba-ad9d-060798e799b7-kube-api-access-4mlwd" pod "bd908f19-bab0-4fba-ad9d-060798e799b7" (UID: "bd908f19-bab0-4fba-ad9d-060798e799b7"). InnerVolumeSpecName "kube-api-access-4mlwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:25.524429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.524414 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-xks-ca" pod "bd908f19-bab0-4fba-ad9d-060798e799b7" (UID: "bd908f19-bab0-4fba-ad9d-060798e799b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:00:25.623241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.623206 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mlwd\" (UniqueName: \"kubernetes.io/projected/bd908f19-bab0-4fba-ad9d-060798e799b7-kube-api-access-4mlwd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:25.623241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.623240 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:25.623241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:25.623249 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd908f19-bab0-4fba-ad9d-060798e799b7-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:26.276417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:26.276388 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" Oct 31 15:00:26.276600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:26.276433 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc" event={"ID":"bd908f19-bab0-4fba-ad9d-060798e799b7","Type":"ContainerDied","Data":"3031a6b0160739546ffef0867745862b5886662a2fffcbfbf9e1aa725b3d3a97"} Oct 31 15:00:26.276600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:26.276502 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3031a6b0160739546ffef0867745862b5886662a2fffcbfbf9e1aa725b3d3a97" Oct 31 15:00:27.504036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:27.503998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:29.323408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:29.323376 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:29.454140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:29.454106 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:29.595085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:29.594965 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:32.302299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:32.302265 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:33.427098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427068 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb"] Oct 31 15:00:33.427526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427512 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bd908f19-bab0-4fba-ad9d-060798e799b7" containerName="revoke-keys" Oct 31 15:00:33.427580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427527 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd908f19-bab0-4fba-ad9d-060798e799b7" containerName="revoke-keys" Oct 31 15:00:33.427580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427562 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:33.427580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427568 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:33.427677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427582 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:33.427677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427587 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8029e38f-28ea-4f72-8624-68bd5bbe8ac2" containerName="cleanup" Oct 31 15:00:33.427677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427659 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfe8640-225e-4bb2-81da-d69f848cd178" containerName="cleanup" Oct 31 15:00:33.427677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427668 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd908f19-bab0-4fba-ad9d-060798e799b7" containerName="revoke-keys" Oct 31 15:00:33.427677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.427678 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7a9e85-b0f5-4f75-8e17-5e54d10ff936" containerName="cleanup" Oct 31 15:00:33.432118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.432100 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.458697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.458673 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:33.486889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.486865 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb"] Oct 31 15:00:33.501112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.501087 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.501235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.501216 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.501317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.501277 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swn2p\" (UniqueName: \"kubernetes.io/projected/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-kube-api-access-swn2p\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.534512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.534479 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:00:33.534512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.534483 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:00:33.534724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.534558 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:00:33.534724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.534571 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:00:33.602278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.602240 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.602438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.602356 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.602438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.602398 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swn2p\" (UniqueName: \"kubernetes.io/projected/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-kube-api-access-swn2p\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.602859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.602842 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.604760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.604742 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.680685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.680592 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swn2p\" (UniqueName: \"kubernetes.io/projected/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-kube-api-access-swn2p\") pod \"maas-api-revoke-sub-ece07229-kfsvb\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:33.744762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:33.744733 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:34.118159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:34.118130 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb"] Oct 31 15:00:34.119260 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:34.119235 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bde3625_e9ab_4d8d_84ae_b25d7a481bf3.slice/crio-524bb2691f72f2f1e55f4d6da61eb17ad1fe24b093c5509a9b96fb96c6edbf17 WatchSource:0}: Error finding container 524bb2691f72f2f1e55f4d6da61eb17ad1fe24b093c5509a9b96fb96c6edbf17: Status 404 returned error can't find the container with id 524bb2691f72f2f1e55f4d6da61eb17ad1fe24b093c5509a9b96fb96c6edbf17 Oct 31 15:00:34.322202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:34.322171 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" event={"ID":"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3","Type":"ContainerStarted","Data":"524bb2691f72f2f1e55f4d6da61eb17ad1fe24b093c5509a9b96fb96c6edbf17"} Oct 31 15:00:35.329264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.329170 4865 generic.go:356] "Generic (PLEG): container finished" podID="9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" containerID="ad0d7fc78a1c8e66937f4af6d6d1fb2111f5fe01743f0406f11ed1e200324085" exitCode=0 Oct 31 15:00:35.329264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.329217 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" event={"ID":"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3","Type":"ContainerDied","Data":"ad0d7fc78a1c8e66937f4af6d6d1fb2111f5fe01743f0406f11ed1e200324085"} Oct 31 15:00:35.710283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.710247 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2"] Oct 31 15:00:35.714954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.714934 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.823067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.823036 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2"] Oct 31 15:00:35.824934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.824904 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.825247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.825205 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88408248-6fd6-4741-9979-f25c6648056b-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.825496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.825449 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.825611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.825595 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88408248-6fd6-4741-9979-f25c6648056b-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.825719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.825707 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.825825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.825811 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.825933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.825921 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.826042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.826015 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.826152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.826137 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-kube-api-access-2sjjs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.926964 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.926999 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88408248-6fd6-4741-9979-f25c6648056b-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927022 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927047 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927080 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927106 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927134 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-kube-api-access-2sjjs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927206 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927243 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88408248-6fd6-4741-9979-f25c6648056b-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927484 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.927873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.927814 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.928430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.928378 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88408248-6fd6-4741-9979-f25c6648056b-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.928561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.928429 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.928561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.928529 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.929788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.929764 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.930891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.930866 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88408248-6fd6-4741-9979-f25c6648056b-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.985565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.985508 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:35.985712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:35.985694 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-kube-api-access-2sjjs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:36.029428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.029393 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:36.461328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.461292 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2"] Oct 31 15:00:36.479109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.479086 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:36.499061 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:36.499031 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88408248_6fd6_4741_9979_f25c6648056b.slice/crio-aac1509f71d8ba4f3c410fc9ef4e716eb16cfe98811908bdf5d06652cd9bd596 WatchSource:0}: Error finding container aac1509f71d8ba4f3c410fc9ef4e716eb16cfe98811908bdf5d06652cd9bd596: Status 404 returned error can't find the container with id aac1509f71d8ba4f3c410fc9ef4e716eb16cfe98811908bdf5d06652cd9bd596 Oct 31 15:00:36.501457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.501422 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:36.501604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.501537 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:36.501604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.501586 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:36.540290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.540269 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-kube-api-access-swn2p\" (UniqueName: \"kubernetes.io/projected/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-kube-api-access-swn2p\") pod \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " Oct 31 15:00:36.540461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.540446 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-openshift-ca\") pod \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " Oct 31 15:00:36.540565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.540544 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-xks-ca\") pod \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\" (UID: \"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3\") " Oct 31 15:00:36.540846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.540816 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-openshift-ca" pod "9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" (UID: "9bde3625-e9ab-4d8d-84ae-b25d7a481bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:36.540965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.540946 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:36.542949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.542924 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-kube-api-access-swn2p" pod "9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" (UID: "9bde3625-e9ab-4d8d-84ae-b25d7a481bf3"). InnerVolumeSpecName "kube-api-access-swn2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:36.543190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.543169 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-xks-ca" pod "9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" (UID: "9bde3625-e9ab-4d8d-84ae-b25d7a481bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:00:36.660277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.660198 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:36.660503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:36.660460 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swn2p\" (UniqueName: \"kubernetes.io/projected/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3-kube-api-access-swn2p\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:37.089487 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:37.089442 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:37.345501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:37.345262 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" event={"ID":"9bde3625-e9ab-4d8d-84ae-b25d7a481bf3","Type":"ContainerDied","Data":"524bb2691f72f2f1e55f4d6da61eb17ad1fe24b093c5509a9b96fb96c6edbf17"} Oct 31 15:00:37.345501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:37.345318 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="524bb2691f72f2f1e55f4d6da61eb17ad1fe24b093c5509a9b96fb96c6edbf17" Oct 31 15:00:37.345501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:37.345270 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb" Oct 31 15:00:37.347692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:37.347650 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" event={"ID":"88408248-6fd6-4741-9979-f25c6648056b","Type":"ContainerStarted","Data":"14005d1aa158119ecd38926258dc93fbacdf1e97ac807dc8b42f73e076ada2a1"} Oct 31 15:00:37.347846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:37.347707 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" event={"ID":"88408248-6fd6-4741-9979-f25c6648056b","Type":"ContainerStarted","Data":"aac1509f71d8ba4f3c410fc9ef4e716eb16cfe98811908bdf5d06652cd9bd596"} Oct 31 15:00:37.578522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:37.577713 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podStartSLOduration=2.5776938 podStartE2EDuration="2.5776938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:37.520946771 +0000 UTC m=+883.140498194" watchObservedRunningTime="2026-10-31 15:00:37.5776938 +0000 UTC m=+883.197245213" Oct 31 15:00:38.029666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:38.029634 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:38.031350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:38.031322 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 15:00:38.031519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:38.031377 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 15:00:39.029696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:39.029661 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 15:00:39.030116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:39.029723 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 15:00:40.029934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:40.029899 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 15:00:40.030453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:40.029956 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 15:00:40.844902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:40.844868 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:00:41.029980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:41.029706 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 15:00:41.029980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:41.029777 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 15:00:42.022532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:42.022494 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:42.029835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:42.029806 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 15:00:42.030086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:42.030061 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 15:00:43.029863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:43.029824 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 15:00:43.030084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:43.029890 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 15:00:44.035240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:44.035194 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:44.035729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:44.035604 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:44.036325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:44.036304 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:00:46.959574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:46.959535 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:47.489917 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:00:47.489884 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:00:49.349652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:49.349623 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:50.378641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:50.378606 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:51.733885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:51.733852 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:54.592452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:54.592422 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:00:54.599111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:54.599087 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:00:54.601406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:54.601385 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:00:54.607296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:54.607278 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:00:55.717504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.717460 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w"] Oct 31 15:00:55.717943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.717930 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" containerName="revoke-keys" Oct 31 15:00:55.717987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.717943 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" containerName="revoke-keys" Oct 31 15:00:55.718028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.718019 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" containerName="revoke-keys" Oct 31 15:00:55.722606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.722587 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.767803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.767775 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc"] Oct 31 15:00:55.771369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.771352 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.794195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794165 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663838ff-9d0b-49d3-8cd4-62bf619754c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794218 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794319 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794343 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794369 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794518 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8k7\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-kube-api-access-sh8k7\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794548 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794573 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.794665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.794653 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.835793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.835762 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w"] Oct 31 15:00:55.895377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895334 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.895601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895413 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.895601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895459 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.895601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895577 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8k7\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-kube-api-access-sh8k7\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.895797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895611 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.895797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895630 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.895797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895669 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.895797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895742 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.895797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895750 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895822 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.896115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895861 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk65k\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-kube-api-access-nk65k\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.896115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895948 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.895976 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.896115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896007 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.896115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896040 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663838ff-9d0b-49d3-8cd4-62bf619754c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896091 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896151 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.896503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896203 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.896503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896250 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896271 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896406 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896702 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.896942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.896908 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663838ff-9d0b-49d3-8cd4-62bf619754c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.898714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.898689 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.898993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.898975 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.935225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.935202 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc"] Oct 31 15:00:55.955813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.955788 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.956204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.956187 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8k7\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-kube-api-access-sh8k7\") pod \"maas-default-gateway-openshift-default-67bf986f99-9cm2w\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:55.997109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997045 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997081 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk65k\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-kube-api-access-nk65k\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997127 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997158 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997231 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997258 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997326 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997383 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997415 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997499 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997631 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997795 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.997845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997814 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.998004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.997955 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.999604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.999587 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:55.999842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:55.999826 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:56.036256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.036220 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:56.056375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.056349 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:56.056538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.056366 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk65k\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-kube-api-access-nk65k\") pod \"maas-default-gateway-openshift-default-67bf986f99-6hmfc\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:56.084613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.084580 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:56.463249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.463222 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w"] Oct 31 15:00:56.464313 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:56.464285 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663838ff_9d0b_49d3_8cd4_62bf619754c8.slice/crio-46b5297380144715c0001bb828d6c54f213a38c639cf1da7f226c2f41543d01f WatchSource:0}: Error finding container 46b5297380144715c0001bb828d6c54f213a38c639cf1da7f226c2f41543d01f: Status 404 returned error can't find the container with id 46b5297380144715c0001bb828d6c54f213a38c639cf1da7f226c2f41543d01f Oct 31 15:00:56.467380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.467345 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:56.467500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.467409 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:56.467500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.467437 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:56.506960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.506931 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:56.518198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.518172 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc"] Oct 31 15:00:56.519970 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:00:56.519942 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7c7719_b0dc_4ccc_a46c_d295a38bdf36.slice/crio-47d5e1cdde5b707376fc24ae70c858f49236ab597fb416f03482f25b8765bb0c WatchSource:0}: Error finding container 47d5e1cdde5b707376fc24ae70c858f49236ab597fb416f03482f25b8765bb0c: Status 404 returned error can't find the container with id 47d5e1cdde5b707376fc24ae70c858f49236ab597fb416f03482f25b8765bb0c Oct 31 15:00:56.522664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.522628 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:56.522728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.522712 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:56.522776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:56.522760 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:00:57.476054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:57.476019 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" event={"ID":"663838ff-9d0b-49d3-8cd4-62bf619754c8","Type":"ContainerStarted","Data":"341f3924b2653e5780a287b6c991aa1bd74ff5bd4e89c8aa68fed365bab84e08"} Oct 31 15:00:57.476660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:57.476642 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" event={"ID":"663838ff-9d0b-49d3-8cd4-62bf619754c8","Type":"ContainerStarted","Data":"46b5297380144715c0001bb828d6c54f213a38c639cf1da7f226c2f41543d01f"} Oct 31 15:00:57.477951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:57.477930 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" event={"ID":"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36","Type":"ContainerStarted","Data":"bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575"} Oct 31 15:00:57.478092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:57.478076 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" event={"ID":"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36","Type":"ContainerStarted","Data":"47d5e1cdde5b707376fc24ae70c858f49236ab597fb416f03482f25b8765bb0c"} Oct 31 15:00:57.738550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:57.736198 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podStartSLOduration=2.736176412 podStartE2EDuration="2.736176412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:57.61379519 +0000 UTC m=+903.233346613" watchObservedRunningTime="2026-10-31 15:00:57.736176412 +0000 UTC m=+903.355727825" Oct 31 15:00:57.755504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:57.753942 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podStartSLOduration=2.75392334 podStartE2EDuration="2.75392334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:57.732882098 +0000 UTC m=+903.352433535" watchObservedRunningTime="2026-10-31 15:00:57.75392334 +0000 UTC m=+903.373474751" Oct 31 15:00:58.054549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:58.054266 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:00:58.054549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:58.054441 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:00:58.054549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:58.054519 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:00:58.085143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:58.085112 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:00:58.086449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:58.086432 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:00:58.086527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:58.086505 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:00:58.818124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:58.818061 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:00:59.039875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:59.039840 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:00:59.040609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:59.040572 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:00:59.085459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:59.085365 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:00:59.085459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:00:59.085427 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:01:00.042954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:00.042917 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:01:00.043611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:00.043575 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:01:00.085314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:00.085285 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:01:00.085592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:00.085567 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:01:01.041495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:01.039879 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:01:01.041495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:01.039943 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:01:01.085440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:01.085403 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:01:01.085841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:01.085453 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:01:02.043733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.043587 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:01:02.043733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.043647 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:01:02.085184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.085145 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:01:02.085333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.085206 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:01:02.184035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.183999 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:02.586007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.585973 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:02.757431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.757394 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:02.897033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:02.896935 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:03.036938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.036907 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:01:03.037100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.036966 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:01:03.084960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.084925 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:01:03.085121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.084990 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:01:03.225397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.225364 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx"] Oct 31 15:01:03.229597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.229576 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.286071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.286043 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx"] Oct 31 15:01:03.334187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.334160 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:01:03.334342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.334196 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:01:03.334342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.334197 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:01:03.334342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.334220 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:01:03.379626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.379601 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.379808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.379789 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.379879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.379839 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtts6\" (UniqueName: \"kubernetes.io/projected/0d537238-fd1d-41b5-8549-e58c22b41fa9-kube-api-access-mtts6\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.480904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.480817 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.480904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.480892 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtts6\" (UniqueName: \"kubernetes.io/projected/0d537238-fd1d-41b5-8549-e58c22b41fa9-kube-api-access-mtts6\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.481112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.480976 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.481629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.481600 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.485541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.485514 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.540024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.539987 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtts6\" (UniqueName: \"kubernetes.io/projected/0d537238-fd1d-41b5-8549-e58c22b41fa9-kube-api-access-mtts6\") pod \"maas-api-revoke-sub-35fcf76c-hq5fx\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.546624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.546599 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:03.946686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:03.946607 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx"] Oct 31 15:01:03.953871 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:01:03.953821 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d537238_fd1d_41b5_8549_e58c22b41fa9.slice/crio-dfeb1983d8d1b68008314bb2be47cda2026489c93dda49aadad5152b845dc5e3 WatchSource:0}: Error finding container dfeb1983d8d1b68008314bb2be47cda2026489c93dda49aadad5152b845dc5e3: Status 404 returned error can't find the container with id dfeb1983d8d1b68008314bb2be47cda2026489c93dda49aadad5152b845dc5e3 Oct 31 15:01:04.036607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:04.036573 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:01:04.036761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:04.036639 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:01:04.084919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:04.084888 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:01:04.085083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:04.084946 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:01:04.570178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:04.570143 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" event={"ID":"0d537238-fd1d-41b5-8549-e58c22b41fa9","Type":"ContainerStarted","Data":"56744f5409c65f05bce581e59e1af344dec12f5f07bf8420fdaefb0bb470e4e8"} Oct 31 15:01:04.570616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:04.570205 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" event={"ID":"0d537238-fd1d-41b5-8549-e58c22b41fa9","Type":"ContainerStarted","Data":"dfeb1983d8d1b68008314bb2be47cda2026489c93dda49aadad5152b845dc5e3"} Oct 31 15:01:04.689915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:04.689863 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" podStartSLOduration=1.6898497940000001 podStartE2EDuration="1.689849794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:04.687170489 +0000 UTC m=+910.306721905" watchObservedRunningTime="2026-10-31 15:01:04.689849794 +0000 UTC m=+910.309401183" Oct 31 15:01:05.037122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:05.037081 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:01:05.037301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:05.037149 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 15:01:05.085808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:05.085770 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 15:01:05.085999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:05.085835 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 15:01:05.588282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:05.588242 4865 generic.go:356] "Generic (PLEG): container finished" podID="0d537238-fd1d-41b5-8549-e58c22b41fa9" containerID="56744f5409c65f05bce581e59e1af344dec12f5f07bf8420fdaefb0bb470e4e8" exitCode=0 Oct 31 15:01:05.588870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:05.588509 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" event={"ID":"0d537238-fd1d-41b5-8549-e58c22b41fa9","Type":"ContainerDied","Data":"56744f5409c65f05bce581e59e1af344dec12f5f07bf8420fdaefb0bb470e4e8"} Oct 31 15:01:06.387756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.386275 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:01:06.387756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.386834 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:01:06.387756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.387684 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:01:06.444036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.444009 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:01:06.444778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.444757 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:01:06.446973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.446950 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:01:06.760289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.760261 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:06.837335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.837288 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-openshift-ca\") pod \"0d537238-fd1d-41b5-8549-e58c22b41fa9\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " Oct 31 15:01:06.837570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.837375 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-xks-ca\") pod \"0d537238-fd1d-41b5-8549-e58c22b41fa9\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " Oct 31 15:01:06.837570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.837427 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d537238-fd1d-41b5-8549-e58c22b41fa9-kube-api-access-mtts6\" (UniqueName: \"kubernetes.io/projected/0d537238-fd1d-41b5-8549-e58c22b41fa9-kube-api-access-mtts6\") pod \"0d537238-fd1d-41b5-8549-e58c22b41fa9\" (UID: \"0d537238-fd1d-41b5-8549-e58c22b41fa9\") " Oct 31 15:01:06.837842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.837812 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-openshift-ca" pod "0d537238-fd1d-41b5-8549-e58c22b41fa9" (UID: "0d537238-fd1d-41b5-8549-e58c22b41fa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:01:06.838052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.838034 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:01:06.840319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.840289 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d537238-fd1d-41b5-8549-e58c22b41fa9-kube-api-access-mtts6" pod "0d537238-fd1d-41b5-8549-e58c22b41fa9" (UID: "0d537238-fd1d-41b5-8549-e58c22b41fa9"). InnerVolumeSpecName "kube-api-access-mtts6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:01:06.842240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.842206 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-xks-ca" pod "0d537238-fd1d-41b5-8549-e58c22b41fa9" (UID: "0d537238-fd1d-41b5-8549-e58c22b41fa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:01:06.939062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.938988 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d537238-fd1d-41b5-8549-e58c22b41fa9-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:01:06.939062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:06.939017 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtts6\" (UniqueName: \"kubernetes.io/projected/0d537238-fd1d-41b5-8549-e58c22b41fa9-kube-api-access-mtts6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:01:07.087772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.087742 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc"] Oct 31 15:01:07.088218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.088204 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0d537238-fd1d-41b5-8549-e58c22b41fa9" containerName="revoke-keys" Oct 31 15:01:07.088218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.088219 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d537238-fd1d-41b5-8549-e58c22b41fa9" containerName="revoke-keys" Oct 31 15:01:07.088328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.088312 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d537238-fd1d-41b5-8549-e58c22b41fa9" containerName="revoke-keys" Oct 31 15:01:07.091600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.091579 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.146011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.145976 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e2aa17aef255\"" Oct 31 15:01:07.146173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146147 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5e686c13c1d4\"" Oct 31 15:01:07.146173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146157 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.146306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146192 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pfxwr\"" Oct 31 15:01:07.146306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146238 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-be9e69b601a9\"" Oct 31 15:01:07.146306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146265 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgcf\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-kube-api-access-mxgcf\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.147001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146421 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-plugins-config-volume\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.147001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146535 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-routing-overlay-volume\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.147001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146549 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 15:01:07.147001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146688 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 15:01:07.147001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146749 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 15:01:07.147001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.146813 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 15:01:07.149156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.148689 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-79e80514dad4\"" Oct 31 15:01:07.151998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.151972 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc"] Oct 31 15:01:07.248017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.247983 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgcf\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-kube-api-access-mxgcf\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.248185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.248082 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-plugins-config-volume\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.248185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.248174 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-routing-overlay-volume\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.248331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.248224 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.248836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.248802 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-routing-overlay-volume\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.248933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.248904 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-plugins-config-volume\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.250910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.250888 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.281633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.281590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:07.320518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.317586 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgcf\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-kube-api-access-mxgcf\") pod \"payload-processing-external-model-7b5d499d94-ht4zc\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.406332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.406303 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:07.601386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.601306 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" Oct 31 15:01:07.601386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.601315 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx" event={"ID":"0d537238-fd1d-41b5-8549-e58c22b41fa9","Type":"ContainerDied","Data":"dfeb1983d8d1b68008314bb2be47cda2026489c93dda49aadad5152b845dc5e3"} Oct 31 15:01:07.601386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.601368 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfeb1983d8d1b68008314bb2be47cda2026489c93dda49aadad5152b845dc5e3" Oct 31 15:01:07.835418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:07.835393 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc"] Oct 31 15:01:07.835993 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:01:07.835964 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7815f242_d478_4f5c_9af3_cd642e0a876b.slice/crio-2640a3db9fd3178312b2212efc6594f23f0679396008e664c2b6f227b054ecee WatchSource:0}: Error finding container 2640a3db9fd3178312b2212efc6594f23f0679396008e664c2b6f227b054ecee: Status 404 returned error can't find the container with id 2640a3db9fd3178312b2212efc6594f23f0679396008e664c2b6f227b054ecee Oct 31 15:01:08.609244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:08.609213 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" event={"ID":"7815f242-d478-4f5c-9af3-cd642e0a876b","Type":"ContainerStarted","Data":"9fe17b116223ad896dac076994fb6f3477ac57ac4b65e9ea6651ba917443856c"} Oct 31 15:01:08.609414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:08.609259 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" event={"ID":"7815f242-d478-4f5c-9af3-cd642e0a876b","Type":"ContainerStarted","Data":"2640a3db9fd3178312b2212efc6594f23f0679396008e664c2b6f227b054ecee"} Oct 31 15:01:08.609414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:08.609340 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:08.725754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:08.725701 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" podStartSLOduration=1.725687136 podStartE2EDuration="1.725687136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:08.725209231 +0000 UTC m=+914.344760680" watchObservedRunningTime="2026-10-31 15:01:08.725687136 +0000 UTC m=+914.345238547" Oct 31 15:01:09.985307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:09.985275 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:01:12.131856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:12.131817 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:17.390137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:17.390108 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:17.507659 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:01:17.507625 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:01:19.617787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:19.617758 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:01:22.097369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:22.097335 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:26.802493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:26.802430 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:32.320922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:32.319033 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:37.151817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:37.151730 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:38.250344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.250318 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k"] Oct 31 15:01:38.255933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.255910 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.354032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.354000 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k"] Oct 31 15:01:38.377691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.377645 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.377870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.377782 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.377870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.377824 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.377870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.377849 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.378010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.377990 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjjd\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-kube-api-access-ctjjd\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.378076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.378063 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.378165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.378153 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.378201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.378173 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.378280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.378266 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479334 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479375 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479444 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479511 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479559 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479590 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479614 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479667 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjjd\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-kube-api-access-ctjjd\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479741 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.479882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.479748 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.480225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.480170 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.480225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.480180 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.480225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.480245 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.480541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.480273 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.481892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.481871 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.482252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.482229 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.540374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.540286 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.540654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.540636 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjjd\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-kube-api-access-ctjjd\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.569052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.569021 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:38.699967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.699937 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:01:38.997434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:38.997410 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k"] Oct 31 15:01:38.998516 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:01:38.998491 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd713974c_1634_4cc0_89c5_cd063ab2a8f8.slice/crio-efa2d99d6f508e8ce97bb1dfce333024feb60e632b81e62564617a08b4332374 WatchSource:0}: Error finding container efa2d99d6f508e8ce97bb1dfce333024feb60e632b81e62564617a08b4332374: Status 404 returned error can't find the container with id efa2d99d6f508e8ce97bb1dfce333024feb60e632b81e62564617a08b4332374 Oct 31 15:01:39.000786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:39.000752 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:01:39.000875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:39.000835 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:01:39.000875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:39.000860 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:01:39.849540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:39.849499 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" event={"ID":"d713974c-1634-4cc0-89c5-cd063ab2a8f8","Type":"ContainerStarted","Data":"41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64"} Oct 31 15:01:39.850019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:39.849555 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" event={"ID":"d713974c-1634-4cc0-89c5-cd063ab2a8f8","Type":"ContainerStarted","Data":"efa2d99d6f508e8ce97bb1dfce333024feb60e632b81e62564617a08b4332374"} Oct 31 15:01:39.978403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:39.978350 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podStartSLOduration=1.978337148 podStartE2EDuration="1.978337148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:39.977717369 +0000 UTC m=+945.597268771" watchObservedRunningTime="2026-10-31 15:01:39.978337148 +0000 UTC m=+945.597888558" Oct 31 15:01:40.569724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:40.569691 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:40.571440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:40.571401 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:01:40.571616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:40.571456 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 15:01:41.570069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:41.570038 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:01:41.570516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:41.570088 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 15:01:41.983090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:41.983056 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:42.569599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:42.569568 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:01:42.569778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:42.569617 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 15:01:43.569406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:43.569372 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:01:43.569856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:43.569426 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 15:01:44.569621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:44.569582 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:01:44.570108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:44.569643 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 15:01:45.887167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:45.886735 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:45.887653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:45.887252 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:45.888154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:45.888137 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:01:46.611759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:46.611732 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:46.745204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:46.745171 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:47.489568 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:01:47.489534 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:01:49.676936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:49.676902 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:50.648528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:50.648458 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:50.788732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:50.788701 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:51.294272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:51.294238 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:54.302340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:54.302283 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:56.015929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:56.015892 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:01:57.466243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:01:57.466205 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:02:00.787333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:00.787293 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:02.258190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:02.258163 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:02.403595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:02.403545 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:05.504626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:05.504595 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:10.202892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:10.202858 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:14.894655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:14.894620 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:17.490227 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:02:17.490186 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:02:18.365205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:18.365177 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:02:19.610582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:19.610549 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:20.072040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:20.072013 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:23.621271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.621236 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t"] Oct 31 15:02:23.625199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.625178 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.715911 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdr9\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-kube-api-access-lmdr9\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.715975 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3f96a-5606-46b4-950c-8353031d73fe-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.716023 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.716072 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3f96a-5606-46b4-950c-8353031d73fe-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.716131 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.716157 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.716201 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.716271 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.716559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.716355 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.734231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.734188 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t"] Oct 31 15:02:23.817928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.817893 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.817969 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818041 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdr9\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-kube-api-access-lmdr9\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818073 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3f96a-5606-46b4-950c-8353031d73fe-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818115 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818164 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3f96a-5606-46b4-950c-8353031d73fe-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818229 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818259 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818302 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818454 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818605 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818656 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818711 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.818861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.818834 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3f96a-5606-46b4-950c-8353031d73fe-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.820774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.820749 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.820910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.820892 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3f96a-5606-46b4-950c-8353031d73fe-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.875584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.875519 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.875584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.875577 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdr9\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-kube-api-access-lmdr9\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:23.939531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:23.939505 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:24.405491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:24.393784 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:24.411714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:24.407583 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t"] Oct 31 15:02:24.423307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:24.423258 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:02:24.423432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:24.423358 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:02:24.423432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:24.423407 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:02:25.140804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:25.140767 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" event={"ID":"4af3f96a-5606-46b4-950c-8353031d73fe","Type":"ContainerStarted","Data":"899fee99a5919fa67ea86df5ecea2ce810039b7711a8297bff327de5a56caf5e"} Oct 31 15:02:25.141222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:25.140846 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" event={"ID":"4af3f96a-5606-46b4-950c-8353031d73fe","Type":"ContainerStarted","Data":"fc805487165b05db14ee1947ed8df643ec31dd6244efe50438a4d68d8dafbbce"} Oct 31 15:02:25.279644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:25.279591 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podStartSLOduration=2.279578366 podStartE2EDuration="2.279578366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:02:25.278231167 +0000 UTC m=+990.897782581" watchObservedRunningTime="2026-10-31 15:02:25.279578366 +0000 UTC m=+990.899129777" Oct 31 15:02:25.940527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:25.940493 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:25.941960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:25.941933 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 15:02:25.942107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:25.941991 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 15:02:26.940481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:26.940441 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 15:02:26.940912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:26.940519 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 15:02:27.939967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:27.939929 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 15:02:27.940160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:27.939996 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 15:02:28.940927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:28.940894 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 15:02:28.941346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:28.940951 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 15:02:29.150902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:29.150872 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:29.940768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:29.940731 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 15:02:29.940944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:29.940790 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 15:02:30.940899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:30.940858 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 15:02:30.941294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:30.940936 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 15:02:31.975630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:31.975298 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:31.976099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:31.975726 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:31.976528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:31.976510 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:02:33.992401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:33.992367 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:38.034893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:38.034862 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:02:38.710405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:38.710372 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:43.423457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:43.423412 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:47.490399 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:02:47.490356 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:02:48.214218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:48.214179 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:50.027949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:50.027916 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:50.640011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:50.639939 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:51.146835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:51.146794 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:52.982624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:52.982590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:54.717112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:54.717086 4865 scope.go:122] "RemoveContainer" containerID="1dcde198efee65f9b74c1c8977e43a6e7af4c168315f56d2f97bbc3d9ad6d287" Oct 31 15:02:54.728847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:54.728822 4865 scope.go:122] "RemoveContainer" containerID="a5972ae7bdbf21e638fc65d9d5082a525356f2b5e7ba64fbea7e84985bb8cba1" Oct 31 15:02:54.737978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:54.737956 4865 scope.go:122] "RemoveContainer" containerID="9773d1b422af314cfd13ea09c3510799e0fadf6b1e93026ac02e00b6a464c638" Oct 31 15:02:57.624300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:57.624267 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:02:59.139734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.139704 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk"] Oct 31 15:02:59.143351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.143335 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147033 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147103 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147172 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147222 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147359 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147401 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147439 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147519 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.147653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.147619 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqw4\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-kube-api-access-7rqw4\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.248727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.248680 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqw4\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-kube-api-access-7rqw4\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.248917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.248758 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.248917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.248793 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.248917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.248825 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.248917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.248863 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.249239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.249215 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.249645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.249621 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.250340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.250262 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.250340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.250313 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.252632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.250370 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.252632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.250404 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.252632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.250710 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.264919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.264891 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.267821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.267789 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.268019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.267960 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.269262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.269236 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.313539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.313513 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.313875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.313848 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqw4\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-kube-api-access-7rqw4\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.346640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.346610 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk"] Oct 31 15:02:59.457186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.457102 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:02:59.881754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.881729 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk"] Oct 31 15:02:59.882210 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:02:59.882184 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e312e76_afbd_49b9_82bf_c7a1d286bde9.slice/crio-c65449a2a1dd8449a9d760f10c7bda8e5777f207a8dc448f4d83bff5e8399a4c WatchSource:0}: Error finding container c65449a2a1dd8449a9d760f10c7bda8e5777f207a8dc448f4d83bff5e8399a4c: Status 404 returned error can't find the container with id c65449a2a1dd8449a9d760f10c7bda8e5777f207a8dc448f4d83bff5e8399a4c Oct 31 15:02:59.884396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.884364 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:02:59.884461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.884432 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:02:59.884562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.884482 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:02:59.917688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:02:59.917662 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:03:00.385106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:00.385069 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" event={"ID":"0e312e76-afbd-49b9-82bf-c7a1d286bde9","Type":"ContainerStarted","Data":"ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24"} Oct 31 15:03:00.385566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:00.385124 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" event={"ID":"0e312e76-afbd-49b9-82bf-c7a1d286bde9","Type":"ContainerStarted","Data":"c65449a2a1dd8449a9d760f10c7bda8e5777f207a8dc448f4d83bff5e8399a4c"} Oct 31 15:03:00.457714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:00.457679 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:03:00.459445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:00.459423 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:03:00.459530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:00.459502 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 15:03:00.513495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:00.510601 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podStartSLOduration=1.510578614 podStartE2EDuration="1.510578614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:00.506406441 +0000 UTC m=+1026.125957853" watchObservedRunningTime="2026-10-31 15:03:00.510578614 +0000 UTC m=+1026.130130027" Oct 31 15:03:01.458282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:01.458245 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:03:01.458716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:01.458314 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 15:03:02.457946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:02.457907 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:03:02.458156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:02.457973 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 15:03:02.545060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:02.545030 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:03.458757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:03.458727 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:03:03.458962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:03.458785 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 15:03:04.457744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:04.457713 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:03:04.458154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:04.457773 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 15:03:05.457988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:05.457909 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:03:05.458369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:05.457981 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 15:03:06.458414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:06.458384 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:03:06.458940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:06.458439 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 31 15:03:07.322622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:07.322589 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:07.669563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:07.666724 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:03:07.669563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:07.667910 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:03:07.670254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:07.670229 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:03:12.026824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:12.026782 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:12.199363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:12.199327 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:15.538721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:15.538693 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:15.665643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:15.665612 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:16.703078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:16.703029 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:17.490370 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:03:17.490335 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:03:21.470978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:21.470938 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:22.866926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:22.866892 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:03:26.193558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:26.193525 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:27.618022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:27.617990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:29.095163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.095123 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw"] Oct 31 15:03:29.099435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.099412 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.156656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.156621 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw"] Oct 31 15:03:29.192172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.192145 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:29.202960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.202935 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"openshift-service-ca.crt\"" Oct 31 15:03:29.203092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.203037 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"kube-root-ca.crt\"" Oct 31 15:03:29.203142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.203104 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"default-dockercfg-w4fm4\"" Oct 31 15:03:29.204254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.204241 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"distinct-2-w4-57815b-kserve-self-signed-certs\"" Oct 31 15:03:29.246026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.245990 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-tmp-dir\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.246408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.246380 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-model-cache\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.246655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.246632 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44g9s\" (UniqueName: \"kubernetes.io/projected/bab3d202-010d-439a-b663-e79536568536-kube-api-access-44g9s\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.246889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.246870 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab3d202-010d-439a-b663-e79536568536-tls-certs\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.247086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.247070 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-dshm\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.247309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.247294 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-home\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349025 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab3d202-010d-439a-b663-e79536568536-tls-certs\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349113 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-dshm\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349220 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-home\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349265 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-tmp-dir\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349318 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-model-cache\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349352 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44g9s\" (UniqueName: \"kubernetes.io/projected/bab3d202-010d-439a-b663-e79536568536-kube-api-access-44g9s\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349698 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-home\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349743 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-model-cache\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.349808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.349751 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-tmp-dir\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.352592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.352573 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-dshm\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.352773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.352754 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab3d202-010d-439a-b663-e79536568536-tls-certs\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.408030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.408000 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44g9s\" (UniqueName: \"kubernetes.io/projected/bab3d202-010d-439a-b663-e79536568536-kube-api-access-44g9s\") pod \"distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.412907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.412871 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:29.799314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:29.799290 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw"] Oct 31 15:03:29.799766 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:03:29.799741 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbab3d202_010d_439a_b663_e79536568536.slice/crio-0839ad87f13f31134057c30dacc681546e011fd7dd9191d1257f9e0954e9c51f WatchSource:0}: Error finding container 0839ad87f13f31134057c30dacc681546e011fd7dd9191d1257f9e0954e9c51f: Status 404 returned error can't find the container with id 0839ad87f13f31134057c30dacc681546e011fd7dd9191d1257f9e0954e9c51f Oct 31 15:03:30.613302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:30.613267 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" event={"ID":"bab3d202-010d-439a-b663-e79536568536","Type":"ContainerStarted","Data":"c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e"} Oct 31 15:03:30.613815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:30.613311 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" event={"ID":"bab3d202-010d-439a-b663-e79536568536","Type":"ContainerStarted","Data":"0839ad87f13f31134057c30dacc681546e011fd7dd9191d1257f9e0954e9c51f"} Oct 31 15:03:30.729949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:30.729897 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podStartSLOduration=1.729884358 podStartE2EDuration="1.729884358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:30.728896881 +0000 UTC m=+1056.348448292" watchObservedRunningTime="2026-10-31 15:03:30.729884358 +0000 UTC m=+1056.349435769" Oct 31 15:03:30.909980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:30.909891 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:31.413241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:31.413211 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:34.546654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.546618 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj"] Oct 31 15:03:34.550898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.550876 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.603531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.599928 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22"] Oct 31 15:03:34.605834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.605807 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.621576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.621544 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.621697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.621660 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nk4l\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-kube-api-access-7nk4l\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.621697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.621687 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.621829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.621702 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e556e968-f2b7-4b38-bc57-e2f6216260e7-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.621829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.621789 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.621910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.621892 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.621957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.621927 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.622022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622004 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622035 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622087 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622110 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622208 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622238 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.622329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622291 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622306 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622372 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.622410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622406 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sj5\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-kube-api-access-w2sj5\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.622499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.622424 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.652441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.652417 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj"] Oct 31 15:03:34.703851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.703818 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22"] Oct 31 15:03:34.723754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.723720 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.723922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.723791 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sj5\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-kube-api-access-w2sj5\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.723922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.723824 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.723922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.723882 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.723944 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nk4l\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-kube-api-access-7nk4l\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.724098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.723971 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.723998 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e556e968-f2b7-4b38-bc57-e2f6216260e7-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724018 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724052 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724096 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724142 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.724349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724164 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.724349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724178 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.724349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724201 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.724916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724594 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724724 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.724916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724799 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724810 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.724916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724838 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724889 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e556e968-f2b7-4b38-bc57-e2f6216260e7-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.724916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724906 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.725348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.724892 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.725348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.725110 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.725348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.725144 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.725348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.725207 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.725348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.725215 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.725655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.725449 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.725655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.725537 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.727060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.727032 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.727686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.727660 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.727871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.727853 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.728195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.728174 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.783169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.783138 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nk4l\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-kube-api-access-7nk4l\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.783324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.783152 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.783324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.783260 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sj5\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-kube-api-access-w2sj5\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.807929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.807862 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:34.865659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.865615 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:34.919337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:34.919306 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:35.301373 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:03:35.301336 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbab9b7d_8c9d_41aa_8016_61e4a20cfc00.slice/crio-11bf1bf65b03cf58e3546dbfb5fe217bd7b5f5ab38689505ee0961df9ef4a2d6 WatchSource:0}: Error finding container 11bf1bf65b03cf58e3546dbfb5fe217bd7b5f5ab38689505ee0961df9ef4a2d6: Status 404 returned error can't find the container with id 11bf1bf65b03cf58e3546dbfb5fe217bd7b5f5ab38689505ee0961df9ef4a2d6 Oct 31 15:03:35.301517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.301374 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj"] Oct 31 15:03:35.303642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.303614 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:35.303715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.303688 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:35.303755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.303718 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:35.352974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.352946 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22"] Oct 31 15:03:35.353749 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:03:35.353726 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode556e968_f2b7_4b38_bc57_e2f6216260e7.slice/crio-68812e23f1b34de69bb4e0c0ae55505217025d82d531665aa2aa6c3a1d1b7617 WatchSource:0}: Error finding container 68812e23f1b34de69bb4e0c0ae55505217025d82d531665aa2aa6c3a1d1b7617: Status 404 returned error can't find the container with id 68812e23f1b34de69bb4e0c0ae55505217025d82d531665aa2aa6c3a1d1b7617 Oct 31 15:03:35.356363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.356330 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:35.356426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.356405 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:35.356463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.356453 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:35.616497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.616404 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:35.647440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.647407 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" event={"ID":"e556e968-f2b7-4b38-bc57-e2f6216260e7","Type":"ContainerStarted","Data":"daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e"} Oct 31 15:03:35.647646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.647459 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" event={"ID":"e556e968-f2b7-4b38-bc57-e2f6216260e7","Type":"ContainerStarted","Data":"68812e23f1b34de69bb4e0c0ae55505217025d82d531665aa2aa6c3a1d1b7617"} Oct 31 15:03:35.649738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.649669 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" event={"ID":"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00","Type":"ContainerStarted","Data":"c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5"} Oct 31 15:03:35.649738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.649721 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" event={"ID":"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00","Type":"ContainerStarted","Data":"11bf1bf65b03cf58e3546dbfb5fe217bd7b5f5ab38689505ee0961df9ef4a2d6"} Oct 31 15:03:35.781173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.778941 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podStartSLOduration=1.7789257630000002 podStartE2EDuration="1.778925763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:35.77644145 +0000 UTC m=+1061.395992860" watchObservedRunningTime="2026-10-31 15:03:35.778925763 +0000 UTC m=+1061.398477174" Oct 31 15:03:35.865937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.865908 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:35.897294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.897200 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podStartSLOduration=1.897184475 podStartE2EDuration="1.897184475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:35.89449066 +0000 UTC m=+1061.514042071" watchObservedRunningTime="2026-10-31 15:03:35.897184475 +0000 UTC m=+1061.516735886" Oct 31 15:03:35.920178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:35.920147 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:36.868403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:36.868367 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:36.868842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:36.868415 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:36.926493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:36.925860 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:36.926493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:36.925921 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:37.866694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:37.866655 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:37.866966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:37.866718 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:37.922495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:37.921572 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:37.922495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:37.921632 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:38.875494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:38.874872 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:38.875494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:38.874950 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:38.926506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:38.924042 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:38.926506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:38.924143 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:39.422553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.421036 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:39.602519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.593093 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:39.604493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.604432 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:39.791497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.790601 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:39.866780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.866145 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:39.866780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.866232 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:39.920023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.919982 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:39.920292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:39.920263 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:40.026782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.026737 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.075785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.075690 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.255498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.254799 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.361556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.357565 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.442496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.437562 4865 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.536622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.536561 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.618368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.615569 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.618368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.616280 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:03:40.794492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.793642 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:40.841905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.831117 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:03:40.875960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.875868 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:40.876202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.876170 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:40.928501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.920973 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:40.936501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.932615 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:40.979997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:40.979960 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:41.027607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.027563 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:41.046171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.046140 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:03:41.079086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.079045 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:41.083495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.079697 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:03:41.301125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.259587 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:41.301125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.259962 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:03:41.398849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.379210 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:41.410068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.410032 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:03:41.601192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.601056 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:41.619109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.618807 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:41.796559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.796517 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:41.891551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.890629 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:41.891551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.890710 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:41.938494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.937803 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:41.938494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:41.937908 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:42.055080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.055040 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:42.085666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.085610 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:42.161553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.138141 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:03:42.260610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.260561 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:03:42.261227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.261193 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:42.425766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.425045 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:42.531646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.531607 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:03:42.531901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.531867 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:42.619216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.619175 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:42.658768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.658735 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:03:42.667493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.663591 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:03:42.680703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.663961 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:03:42.680703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.680604 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:03:42.756811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.756778 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:03:42.762495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.758630 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:03:42.762495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.761325 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:03:42.766764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.766745 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:03:42.866581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.866544 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:42.866770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.866619 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:42.937036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.936632 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:42.937036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:42.936703 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:43.866730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:43.866691 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:03:43.867125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:43.866753 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 15:03:43.920608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:43.920572 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 15:03:43.920774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:43.920642 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 15:03:44.871162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:44.871128 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:44.871571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:44.871505 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:44.872364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:44.872340 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:03:45.244924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:45.244857 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:45.245884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:45.245858 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:45.246997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:45.246975 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 15:03:45.710778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:45.710744 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:47.489965 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:03:47.489919 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:03:49.428834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:49.428803 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:49.446603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:49.446573 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:03:49.475423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:49.475388 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:50.379593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:50.379559 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:50.517284 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:50.517248 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:51.498955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:51.498923 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:54.326851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.326818 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7"] Oct 31 15:03:54.330709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.330690 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.374938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.374901 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.374944 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.375093 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.375198 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.375238 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec12d57-7084-41cc-8d01-aafab86143d5-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.375262 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.375383 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4p4\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-kube-api-access-hv4p4\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.375420 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec12d57-7084-41cc-8d01-aafab86143d5-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.375455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.375442 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.470674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.470578 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7"] Oct 31 15:03:54.476577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476491 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.476577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476543 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476608 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476675 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476721 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec12d57-7084-41cc-8d01-aafab86143d5-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476752 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476818 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4p4\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-kube-api-access-hv4p4\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476855 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec12d57-7084-41cc-8d01-aafab86143d5-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.476885 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.477370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.477334 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.478068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.478041 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec12d57-7084-41cc-8d01-aafab86143d5-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.478228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.478197 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.478907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.478540 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.478907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.478627 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.480377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.480351 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec12d57-7084-41cc-8d01-aafab86143d5-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.480659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.480638 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.535946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.535915 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.536163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.536135 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4p4\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-kube-api-access-hv4p4\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:54.644671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:54.644607 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:55.075181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:55.075149 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7"] Oct 31 15:03:55.075646 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:03:55.075616 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec12d57_7084_41cc_8d01_aafab86143d5.slice/crio-4433ec042e16221921758c5a4576a8d5116cf372bfd3c65dd241842097783c49 WatchSource:0}: Error finding container 4433ec042e16221921758c5a4576a8d5116cf372bfd3c65dd241842097783c49: Status 404 returned error can't find the container with id 4433ec042e16221921758c5a4576a8d5116cf372bfd3c65dd241842097783c49 Oct 31 15:03:55.078126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:55.078091 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:55.078213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:55.078162 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:55.078213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:55.078191 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:03:55.398897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:55.398794 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:03:56.044884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.044849 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" event={"ID":"cec12d57-7084-41cc-8d01-aafab86143d5","Type":"ContainerStarted","Data":"b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2"} Oct 31 15:03:56.045111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.044898 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" event={"ID":"cec12d57-7084-41cc-8d01-aafab86143d5","Type":"ContainerStarted","Data":"4433ec042e16221921758c5a4576a8d5116cf372bfd3c65dd241842097783c49"} Oct 31 15:03:56.062532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.062501 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7"] Oct 31 15:03:56.066573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.066551 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.124280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.124239 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6v8z\" (UniqueName: \"kubernetes.io/projected/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-kube-api-access-c6v8z\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.124518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.124486 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.124590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.124542 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.124590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.124556 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7"] Oct 31 15:03:56.151970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.151945 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:03:56.152135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.152075 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:03:56.152135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.152099 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:03:56.152268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.152156 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:03:56.226001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.225958 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6v8z\" (UniqueName: \"kubernetes.io/projected/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-kube-api-access-c6v8z\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.226179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.226125 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.226179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.226170 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.226872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.226845 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.228686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.228667 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.283808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.283786 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6v8z\" (UniqueName: \"kubernetes.io/projected/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-kube-api-access-c6v8z\") pod \"maas-api-revoke-sub-accf6735-wg6d7\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.335559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.335437 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podStartSLOduration=2.335423721 podStartE2EDuration="2.335423721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:56.218803045 +0000 UTC m=+1081.838354456" watchObservedRunningTime="2026-10-31 15:03:56.335423721 +0000 UTC m=+1081.954975186" Oct 31 15:03:56.381195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.381168 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:56.645367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.645275 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:03:56.646678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.646657 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 15:03:56.646761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.646701 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 15:03:56.785804 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:03:56.785777 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690a9272_7b7b_4bbe_9bc1_22f84b0002d5.slice/crio-cdac654036feb1304ef86bb03a2db8d508cf0f2c78b06a1b4d535da9e4c7317a WatchSource:0}: Error finding container cdac654036feb1304ef86bb03a2db8d508cf0f2c78b06a1b4d535da9e4c7317a: Status 404 returned error can't find the container with id cdac654036feb1304ef86bb03a2db8d508cf0f2c78b06a1b4d535da9e4c7317a Oct 31 15:03:56.789422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:56.789395 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7"] Oct 31 15:03:57.052544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:57.052517 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" event={"ID":"690a9272-7b7b-4bbe-9bc1-22f84b0002d5","Type":"ContainerStarted","Data":"cdac654036feb1304ef86bb03a2db8d508cf0f2c78b06a1b4d535da9e4c7317a"} Oct 31 15:03:57.645411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:57.645377 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 15:03:57.645847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:57.645436 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 15:03:58.060291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:58.060256 4865 generic.go:356] "Generic (PLEG): container finished" podID="690a9272-7b7b-4bbe-9bc1-22f84b0002d5" containerID="5a2c6a74ea0c70e4a17d150fb785fdb5ee81668364825f15d3e337fadfedbc54" exitCode=0 Oct 31 15:03:58.060506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:58.060335 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" event={"ID":"690a9272-7b7b-4bbe-9bc1-22f84b0002d5","Type":"ContainerDied","Data":"5a2c6a74ea0c70e4a17d150fb785fdb5ee81668364825f15d3e337fadfedbc54"} Oct 31 15:03:58.644966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:58.644929 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 15:03:58.645131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:58.644990 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 15:03:59.264560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.264531 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:03:59.365013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.364934 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-openshift-ca\") pod \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " Oct 31 15:03:59.365013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.364969 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-kube-api-access-c6v8z\" (UniqueName: \"kubernetes.io/projected/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-kube-api-access-c6v8z\") pod \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " Oct 31 15:03:59.365207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.365031 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-xks-ca\") pod \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\" (UID: \"690a9272-7b7b-4bbe-9bc1-22f84b0002d5\") " Oct 31 15:03:59.365279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.365243 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-openshift-ca" pod "690a9272-7b7b-4bbe-9bc1-22f84b0002d5" (UID: "690a9272-7b7b-4bbe-9bc1-22f84b0002d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:03:59.365420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.365384 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:03:59.367382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.367361 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-xks-ca" pod "690a9272-7b7b-4bbe-9bc1-22f84b0002d5" (UID: "690a9272-7b7b-4bbe-9bc1-22f84b0002d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:03:59.367581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.367554 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-kube-api-access-c6v8z" pod "690a9272-7b7b-4bbe-9bc1-22f84b0002d5" (UID: "690a9272-7b7b-4bbe-9bc1-22f84b0002d5"). InnerVolumeSpecName "kube-api-access-c6v8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:03:59.466400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.466361 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:03:59.466400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.466391 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6v8z\" (UniqueName: \"kubernetes.io/projected/690a9272-7b7b-4bbe-9bc1-22f84b0002d5-kube-api-access-c6v8z\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:03:59.613494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.609931 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg"] Oct 31 15:03:59.613494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.610831 4865 cpu_manager.go:414] "containerMap: removing container" podUID="690a9272-7b7b-4bbe-9bc1-22f84b0002d5" containerName="revoke-keys" Oct 31 15:03:59.613494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.610850 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690a9272-7b7b-4bbe-9bc1-22f84b0002d5" containerName="revoke-keys" Oct 31 15:03:59.613494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.610980 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="690a9272-7b7b-4bbe-9bc1-22f84b0002d5" containerName="revoke-keys" Oct 31 15:03:59.616119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.616052 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.645643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.645609 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 15:03:59.645778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.645662 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 15:03:59.655724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.655698 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg"] Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.669140 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.669250 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e6edf-82fb-4819-aa04-21f0db05769b-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.669314 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.669348 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.669411 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.669456 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrr8k\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-kube-api-access-nrr8k\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.669650 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.670137 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.670486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.670165 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771341 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771381 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771417 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771440 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrr8k\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-kube-api-access-nrr8k\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771558 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771579 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771872 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771872 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.771909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771871 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.772077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771934 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.772077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.771953 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.772077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.772048 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e6edf-82fb-4819-aa04-21f0db05769b-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.772230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.772098 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.772707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.772685 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e6edf-82fb-4819-aa04-21f0db05769b-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.774363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.774343 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.774493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.774393 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.829931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.829893 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.830291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.830274 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrr8k\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-kube-api-access-nrr8k\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:03:59.935598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:03:59.935514 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:04:00.080274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.080230 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" event={"ID":"690a9272-7b7b-4bbe-9bc1-22f84b0002d5","Type":"ContainerDied","Data":"cdac654036feb1304ef86bb03a2db8d508cf0f2c78b06a1b4d535da9e4c7317a"} Oct 31 15:04:00.080433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.080297 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdac654036feb1304ef86bb03a2db8d508cf0f2c78b06a1b4d535da9e4c7317a" Oct 31 15:04:00.080433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.080365 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7" Oct 31 15:04:00.367028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.366982 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg"] Oct 31 15:04:00.367339 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:00.367202 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e6edf_82fb_4819_aa04_21f0db05769b.slice/crio-67e75b669639ddd50b37a3b46152b390310b80729dd5e1fc3a1f8e24376720c9 WatchSource:0}: Error finding container 67e75b669639ddd50b37a3b46152b390310b80729dd5e1fc3a1f8e24376720c9: Status 404 returned error can't find the container with id 67e75b669639ddd50b37a3b46152b390310b80729dd5e1fc3a1f8e24376720c9 Oct 31 15:04:00.369970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.369936 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:04:00.370035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.370018 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:04:00.370077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.370066 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:04:00.444300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.444268 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:00.652293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.651839 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 15:04:00.652293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:00.651894 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 15:04:01.088672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.088641 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" event={"ID":"c10e6edf-82fb-4819-aa04-21f0db05769b","Type":"ContainerStarted","Data":"b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3"} Oct 31 15:04:01.088847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.088689 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" event={"ID":"c10e6edf-82fb-4819-aa04-21f0db05769b","Type":"ContainerStarted","Data":"67e75b669639ddd50b37a3b46152b390310b80729dd5e1fc3a1f8e24376720c9"} Oct 31 15:04:01.209093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.209040 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podStartSLOduration=2.209021844 podStartE2EDuration="2.209021844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:01.206979699 +0000 UTC m=+1086.826531122" watchObservedRunningTime="2026-10-31 15:04:01.209021844 +0000 UTC m=+1086.828573259" Oct 31 15:04:01.645488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.645436 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 15:04:01.645997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.645516 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 15:04:01.936218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.936115 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:04:01.938068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.938036 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 15:04:01.938221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:01.938091 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 15:04:02.549527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:02.549492 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:04:02.936045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:02.935949 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 15:04:02.936045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:02.936002 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 15:04:03.143549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:03.143510 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:04:03.144338 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:03.144315 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:04:03.145168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:03.145146 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:04:04.051494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:04.000402 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 15:04:04.053134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:04.052730 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 15:04:04.580891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:04.580843 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:04.944548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:04.944251 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 15:04:04.944548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:04.944308 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 15:04:05.939927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:05.939888 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 15:04:05.940604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:05.940568 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 15:04:05.982495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:05.980670 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:06.936692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:06.936659 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 15:04:06.936899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:06.936735 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 15:04:07.936812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:07.936774 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 15:04:07.937200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:07.936832 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 15:04:09.633237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:09.633205 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:04:09.633714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:09.633645 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:04:09.634788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:09.634770 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:04:10.895704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:10.895666 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:11.001927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.001883 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt"] Oct 31 15:04:11.009393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.009364 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.047429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.047390 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt"] Oct 31 15:04:11.093777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.093736 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:04:11.093777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.093772 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:04:11.093961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.093789 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:04:11.110454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.110433 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.110589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.110554 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.110722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.110705 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mln9s\" (UniqueName: \"kubernetes.io/projected/dee3f5c5-b61d-44bf-9e78-276b57eaf159-kube-api-access-mln9s\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.114396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.114380 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:04:11.211770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.211747 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mln9s\" (UniqueName: \"kubernetes.io/projected/dee3f5c5-b61d-44bf-9e78-276b57eaf159-kube-api-access-mln9s\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.211904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.211849 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.211979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.211914 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.212520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.212463 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.214437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.214417 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.269981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.269959 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mln9s\" (UniqueName: \"kubernetes.io/projected/dee3f5c5-b61d-44bf-9e78-276b57eaf159-kube-api-access-mln9s\") pod \"maas-api-revoke-sub-eb1c88a6-xlxzt\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.340032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.340005 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:11.711137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:11.711110 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt"] Oct 31 15:04:11.711576 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:11.711554 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee3f5c5_b61d_44bf_9e78_276b57eaf159.slice/crio-d6528bf35b4161775d269f108ac95f19e74da483bc0f4212b9b715f29e13038e WatchSource:0}: Error finding container d6528bf35b4161775d269f108ac95f19e74da483bc0f4212b9b715f29e13038e: Status 404 returned error can't find the container with id d6528bf35b4161775d269f108ac95f19e74da483bc0f4212b9b715f29e13038e Oct 31 15:04:12.314572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.314464 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" event={"ID":"dee3f5c5-b61d-44bf-9e78-276b57eaf159","Type":"ContainerStarted","Data":"2bb58c511d13acfd1102168d921d5644f6ac7a6448f95710bfcfdce9eacf5752"} Oct 31 15:04:12.314572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.314546 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" event={"ID":"dee3f5c5-b61d-44bf-9e78-276b57eaf159","Type":"ContainerStarted","Data":"d6528bf35b4161775d269f108ac95f19e74da483bc0f4212b9b715f29e13038e"} Oct 31 15:04:12.422212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.422166 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k"] Oct 31 15:04:12.425962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.425939 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.447431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.447389 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" podStartSLOduration=2.447377833 podStartE2EDuration="2.447377833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:12.429351462 +0000 UTC m=+1098.048902878" watchObservedRunningTime="2026-10-31 15:04:12.447377833 +0000 UTC m=+1098.066929243" Oct 31 15:04:12.458784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.458755 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k"] Oct 31 15:04:12.528023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.527989 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.528238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.528218 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.528304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.528262 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5s8\" (UniqueName: \"kubernetes.io/projected/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-kube-api-access-gg5s8\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.629502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.629416 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.629502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.629459 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5s8\" (UniqueName: \"kubernetes.io/projected/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-kube-api-access-gg5s8\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.629716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.629529 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.630167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.630140 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.632100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.632080 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.693691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.693666 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5s8\" (UniqueName: \"kubernetes.io/projected/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-kube-api-access-gg5s8\") pod \"maas-api-revoke-sub-da4d2ca1-w8h6k\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:12.741789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:12.741758 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:13.120961 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:13.120936 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73341bea_aa9b_47e4_9b6c_0b34e29b9f47.slice/crio-f66d37d3c741b0ff1e5dc0ced067f18d52ab87cc365082473eab32057148005d WatchSource:0}: Error finding container f66d37d3c741b0ff1e5dc0ced067f18d52ab87cc365082473eab32057148005d: Status 404 returned error can't find the container with id f66d37d3c741b0ff1e5dc0ced067f18d52ab87cc365082473eab32057148005d Oct 31 15:04:13.121463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:13.121437 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k"] Oct 31 15:04:13.324701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:13.324661 4865 generic.go:356] "Generic (PLEG): container finished" podID="dee3f5c5-b61d-44bf-9e78-276b57eaf159" containerID="2bb58c511d13acfd1102168d921d5644f6ac7a6448f95710bfcfdce9eacf5752" exitCode=0 Oct 31 15:04:13.325112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:13.324818 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" event={"ID":"dee3f5c5-b61d-44bf-9e78-276b57eaf159","Type":"ContainerDied","Data":"2bb58c511d13acfd1102168d921d5644f6ac7a6448f95710bfcfdce9eacf5752"} Oct 31 15:04:13.326917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:13.326887 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" event={"ID":"73341bea-aa9b-47e4-9b6c-0b34e29b9f47","Type":"ContainerStarted","Data":"f66d37d3c741b0ff1e5dc0ced067f18d52ab87cc365082473eab32057148005d"} Oct 31 15:04:14.334583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.334500 4865 generic.go:356] "Generic (PLEG): container finished" podID="73341bea-aa9b-47e4-9b6c-0b34e29b9f47" containerID="19f82563988d15e0e026e575bdf8dcaeb047ad73c84e0d98ef37160fb7a10b04" exitCode=0 Oct 31 15:04:14.334941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.334631 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" event={"ID":"73341bea-aa9b-47e4-9b6c-0b34e29b9f47","Type":"ContainerDied","Data":"19f82563988d15e0e026e575bdf8dcaeb047ad73c84e0d98ef37160fb7a10b04"} Oct 31 15:04:14.496917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.496891 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:14.550892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.550867 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee3f5c5-b61d-44bf-9e78-276b57eaf159-kube-api-access-mln9s\" (UniqueName: \"kubernetes.io/projected/dee3f5c5-b61d-44bf-9e78-276b57eaf159-kube-api-access-mln9s\") pod \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " Oct 31 15:04:14.551054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.550956 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-xks-ca\") pod \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " Oct 31 15:04:14.551109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.551096 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-openshift-ca\") pod \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\" (UID: \"dee3f5c5-b61d-44bf-9e78-276b57eaf159\") " Oct 31 15:04:14.551512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.551452 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-openshift-ca" pod "dee3f5c5-b61d-44bf-9e78-276b57eaf159" (UID: "dee3f5c5-b61d-44bf-9e78-276b57eaf159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:14.551660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.551633 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:14.553344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.553322 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-xks-ca" pod "dee3f5c5-b61d-44bf-9e78-276b57eaf159" (UID: "dee3f5c5-b61d-44bf-9e78-276b57eaf159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:14.553429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.553320 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee3f5c5-b61d-44bf-9e78-276b57eaf159-kube-api-access-mln9s" pod "dee3f5c5-b61d-44bf-9e78-276b57eaf159" (UID: "dee3f5c5-b61d-44bf-9e78-276b57eaf159"). InnerVolumeSpecName "kube-api-access-mln9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:14.652568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.652489 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mln9s\" (UniqueName: \"kubernetes.io/projected/dee3f5c5-b61d-44bf-9e78-276b57eaf159-kube-api-access-mln9s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:14.652568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:14.652519 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee3f5c5-b61d-44bf-9e78-276b57eaf159-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:15.342032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.342001 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" Oct 31 15:04:15.342441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.342033 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt" event={"ID":"dee3f5c5-b61d-44bf-9e78-276b57eaf159","Type":"ContainerDied","Data":"d6528bf35b4161775d269f108ac95f19e74da483bc0f4212b9b715f29e13038e"} Oct 31 15:04:15.342441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.342078 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6528bf35b4161775d269f108ac95f19e74da483bc0f4212b9b715f29e13038e" Oct 31 15:04:15.503131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.503105 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:15.564073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.564037 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-kube-api-access-gg5s8\" (UniqueName: \"kubernetes.io/projected/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-kube-api-access-gg5s8\") pod \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " Oct 31 15:04:15.564251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.564237 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-openshift-ca\") pod \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " Oct 31 15:04:15.564316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.564292 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-xks-ca\") pod \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\" (UID: \"73341bea-aa9b-47e4-9b6c-0b34e29b9f47\") " Oct 31 15:04:15.564617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.564591 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-openshift-ca" pod "73341bea-aa9b-47e4-9b6c-0b34e29b9f47" (UID: "73341bea-aa9b-47e4-9b6c-0b34e29b9f47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:15.564985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.564967 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:15.566321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.566302 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-kube-api-access-gg5s8" pod "73341bea-aa9b-47e4-9b6c-0b34e29b9f47" (UID: "73341bea-aa9b-47e4-9b6c-0b34e29b9f47"). InnerVolumeSpecName "kube-api-access-gg5s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:15.566886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.566865 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-xks-ca" pod "73341bea-aa9b-47e4-9b6c-0b34e29b9f47" (UID: "73341bea-aa9b-47e4-9b6c-0b34e29b9f47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:15.611500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.611395 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:15.677494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.666065 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:15.677494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:15.666102 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5s8\" (UniqueName: \"kubernetes.io/projected/73341bea-aa9b-47e4-9b6c-0b34e29b9f47-kube-api-access-gg5s8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:16.350660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:16.350632 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" Oct 31 15:04:16.351040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:16.350659 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k" event={"ID":"73341bea-aa9b-47e4-9b6c-0b34e29b9f47","Type":"ContainerDied","Data":"f66d37d3c741b0ff1e5dc0ced067f18d52ab87cc365082473eab32057148005d"} Oct 31 15:04:16.351040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:16.350705 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66d37d3c741b0ff1e5dc0ced067f18d52ab87cc365082473eab32057148005d" Oct 31 15:04:17.490658 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:04:17.490609 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:04:20.307327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:20.307298 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:23.670643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:23.670555 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:23.785842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:23.785796 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:23.787639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:23.787606 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:04:23.787953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:23.787554 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:23.853496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:23.840573 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:24.438613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:24.438566 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:24.439245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:24.439213 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:04:24.621568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:24.621529 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:04:24.676932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:24.676887 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:24.677884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:24.677860 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:04:25.312030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:25.312000 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:04:25.608210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:25.608128 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:28.051676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.051644 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444"] Oct 31 15:04:28.102668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.102638 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a48b23-nn444"] Oct 31 15:04:28.496424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.496386 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd6712d-b0cf-48fc-9fb2-c226efbf92bd" path="/var/lib/kubelet/pods/2cd6712d-b0cf-48fc-9fb2-c226efbf92bd/volumes" Oct 31 15:04:28.719737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.719705 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5"] Oct 31 15:04:28.720268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.720254 4865 cpu_manager.go:414] "containerMap: removing container" podUID="dee3f5c5-b61d-44bf-9e78-276b57eaf159" containerName="revoke-keys" Oct 31 15:04:28.720313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.720269 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee3f5c5-b61d-44bf-9e78-276b57eaf159" containerName="revoke-keys" Oct 31 15:04:28.720313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.720308 4865 cpu_manager.go:414] "containerMap: removing container" podUID="73341bea-aa9b-47e4-9b6c-0b34e29b9f47" containerName="revoke-keys" Oct 31 15:04:28.720380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.720313 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73341bea-aa9b-47e4-9b6c-0b34e29b9f47" containerName="revoke-keys" Oct 31 15:04:28.720417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.720407 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee3f5c5-b61d-44bf-9e78-276b57eaf159" containerName="revoke-keys" Oct 31 15:04:28.720452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.720426 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="73341bea-aa9b-47e4-9b6c-0b34e29b9f47" containerName="revoke-keys" Oct 31 15:04:28.725031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.725015 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.750884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.750816 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.750884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.750857 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzmdj\" (UniqueName: \"kubernetes.io/projected/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-kube-api-access-xzmdj\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.751057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.750890 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.780349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.780320 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5"] Oct 31 15:04:28.827449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.827417 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:04:28.827449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.827451 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:04:28.827676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.827501 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:04:28.829628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.829611 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:04:28.851762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.851738 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzmdj\" (UniqueName: \"kubernetes.io/projected/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-kube-api-access-xzmdj\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.851918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.851772 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.851986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.851937 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.852417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.852399 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.854300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.854281 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:28.909713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:28.909688 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzmdj\" (UniqueName: \"kubernetes.io/projected/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-kube-api-access-xzmdj\") pod \"maas-api-revoke-sub-b2d4ea60-t7nw5\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:29.038582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:29.038507 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:29.411270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:29.411241 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5"] Oct 31 15:04:29.411851 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:29.411824 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a0d1d4_d59e_4c69_8920_bd81dc3a851e.slice/crio-b8aeadb767b2268aa62d869e93cfb7ac557ff8ce7c292386d27d5a2cf4d6458d WatchSource:0}: Error finding container b8aeadb767b2268aa62d869e93cfb7ac557ff8ce7c292386d27d5a2cf4d6458d: Status 404 returned error can't find the container with id b8aeadb767b2268aa62d869e93cfb7ac557ff8ce7c292386d27d5a2cf4d6458d Oct 31 15:04:29.603446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:29.603412 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" event={"ID":"55a0d1d4-d59e-4c69-8920-bd81dc3a851e","Type":"ContainerStarted","Data":"b8aeadb767b2268aa62d869e93cfb7ac557ff8ce7c292386d27d5a2cf4d6458d"} Oct 31 15:04:30.318381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:30.318313 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:30.612373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:30.612292 4865 generic.go:356] "Generic (PLEG): container finished" podID="55a0d1d4-d59e-4c69-8920-bd81dc3a851e" containerID="2587ae9f5c62f12eaf058d68e6516dc71c22be1bda23d7ccb2a8f5c08a230441" exitCode=0 Oct 31 15:04:30.612751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:30.612372 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" event={"ID":"55a0d1d4-d59e-4c69-8920-bd81dc3a851e","Type":"ContainerDied","Data":"2587ae9f5c62f12eaf058d68e6516dc71c22be1bda23d7ccb2a8f5c08a230441"} Oct 31 15:04:31.774948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.774923 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:31.806629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.806599 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-xks-ca\") pod \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " Oct 31 15:04:31.806777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.806737 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-openshift-ca\") pod \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " Oct 31 15:04:31.806835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.806815 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-kube-api-access-xzmdj\" (UniqueName: \"kubernetes.io/projected/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-kube-api-access-xzmdj\") pod \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\" (UID: \"55a0d1d4-d59e-4c69-8920-bd81dc3a851e\") " Oct 31 15:04:31.807027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.806999 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-openshift-ca" pod "55a0d1d4-d59e-4c69-8920-bd81dc3a851e" (UID: "55a0d1d4-d59e-4c69-8920-bd81dc3a851e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:31.807313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.807286 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:31.809249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.809228 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-xks-ca" pod "55a0d1d4-d59e-4c69-8920-bd81dc3a851e" (UID: "55a0d1d4-d59e-4c69-8920-bd81dc3a851e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:31.809446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.809426 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-kube-api-access-xzmdj" pod "55a0d1d4-d59e-4c69-8920-bd81dc3a851e" (UID: "55a0d1d4-d59e-4c69-8920-bd81dc3a851e"). InnerVolumeSpecName "kube-api-access-xzmdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:31.908317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.908239 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzmdj\" (UniqueName: \"kubernetes.io/projected/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-kube-api-access-xzmdj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:31.908317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:31.908267 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a0d1d4-d59e-4c69-8920-bd81dc3a851e-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:32.627894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:32.627867 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" Oct 31 15:04:32.628084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:32.627905 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5" event={"ID":"55a0d1d4-d59e-4c69-8920-bd81dc3a851e","Type":"ContainerDied","Data":"b8aeadb767b2268aa62d869e93cfb7ac557ff8ce7c292386d27d5a2cf4d6458d"} Oct 31 15:04:32.628084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:32.627980 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8aeadb767b2268aa62d869e93cfb7ac557ff8ce7c292386d27d5a2cf4d6458d" Oct 31 15:04:35.311511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:35.297909 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:36.750136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.750109 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg"] Oct 31 15:04:36.750857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.750836 4865 cpu_manager.go:414] "containerMap: removing container" podUID="55a0d1d4-d59e-4c69-8920-bd81dc3a851e" containerName="revoke-keys" Oct 31 15:04:36.750964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.750859 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a0d1d4-d59e-4c69-8920-bd81dc3a851e" containerName="revoke-keys" Oct 31 15:04:36.751070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.751056 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a0d1d4-d59e-4c69-8920-bd81dc3a851e" containerName="revoke-keys" Oct 31 15:04:36.754305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.754284 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.810254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.810225 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg"] Oct 31 15:04:36.832113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.832078 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6ld\" (UniqueName: \"kubernetes.io/projected/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-kube-api-access-sm6ld\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.832361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.832341 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.832425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.832374 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.856715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.856685 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:04:36.856715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.856705 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:04:36.856895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.856819 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:04:36.858321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.858304 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:04:36.933785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.933758 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.933785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.933787 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.933972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.933858 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6ld\" (UniqueName: \"kubernetes.io/projected/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-kube-api-access-sm6ld\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.934415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.934386 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.936206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.936189 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:36.992150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:36.992121 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6ld\" (UniqueName: \"kubernetes.io/projected/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-kube-api-access-sm6ld\") pod \"maas-api-revoke-sub-1156bf32-p6rxg\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:37.068158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:37.068097 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:37.445951 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:37.445920 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4828559a_e3f6_4b7f_8ecd_e006a97b0bba.slice/crio-da8943e5db4e2c246b45e84ae2bf34fdb6505eda6cdb6ea235d1ecf62d21c93c WatchSource:0}: Error finding container da8943e5db4e2c246b45e84ae2bf34fdb6505eda6cdb6ea235d1ecf62d21c93c: Status 404 returned error can't find the container with id da8943e5db4e2c246b45e84ae2bf34fdb6505eda6cdb6ea235d1ecf62d21c93c Oct 31 15:04:37.446664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:37.446640 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg"] Oct 31 15:04:37.710542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:37.710509 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" event={"ID":"4828559a-e3f6-4b7f-8ecd-e006a97b0bba","Type":"ContainerStarted","Data":"da8943e5db4e2c246b45e84ae2bf34fdb6505eda6cdb6ea235d1ecf62d21c93c"} Oct 31 15:04:38.718451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:38.718418 4865 generic.go:356] "Generic (PLEG): container finished" podID="4828559a-e3f6-4b7f-8ecd-e006a97b0bba" containerID="8f9db0dcf3d1b82854aac6998a89c7b2c67a4d65da0033d95729849d52a63a00" exitCode=0 Oct 31 15:04:38.718866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:38.718509 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" event={"ID":"4828559a-e3f6-4b7f-8ecd-e006a97b0bba","Type":"ContainerDied","Data":"8f9db0dcf3d1b82854aac6998a89c7b2c67a4d65da0033d95729849d52a63a00"} Oct 31 15:04:39.891351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:39.891328 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:40.076833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.076751 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-openshift-ca\") pod \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " Oct 31 15:04:40.076833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.076819 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-xks-ca\") pod \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " Oct 31 15:04:40.077076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.076846 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-kube-api-access-sm6ld\" (UniqueName: \"kubernetes.io/projected/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-kube-api-access-sm6ld\") pod \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\" (UID: \"4828559a-e3f6-4b7f-8ecd-e006a97b0bba\") " Oct 31 15:04:40.077137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.077114 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-openshift-ca" pod "4828559a-e3f6-4b7f-8ecd-e006a97b0bba" (UID: "4828559a-e3f6-4b7f-8ecd-e006a97b0bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:40.077408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.077386 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:40.079279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.079237 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-xks-ca" pod "4828559a-e3f6-4b7f-8ecd-e006a97b0bba" (UID: "4828559a-e3f6-4b7f-8ecd-e006a97b0bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:40.079600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.079563 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-kube-api-access-sm6ld" pod "4828559a-e3f6-4b7f-8ecd-e006a97b0bba" (UID: "4828559a-e3f6-4b7f-8ecd-e006a97b0bba"). InnerVolumeSpecName "kube-api-access-sm6ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:40.115266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.115241 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:40.193055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.192985 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:40.193055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.193020 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6ld\" (UniqueName: \"kubernetes.io/projected/4828559a-e3f6-4b7f-8ecd-e006a97b0bba-kube-api-access-sm6ld\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:40.633312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.633281 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:40.749272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.749231 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" event={"ID":"4828559a-e3f6-4b7f-8ecd-e006a97b0bba","Type":"ContainerDied","Data":"da8943e5db4e2c246b45e84ae2bf34fdb6505eda6cdb6ea235d1ecf62d21c93c"} Oct 31 15:04:40.749575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.749554 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da8943e5db4e2c246b45e84ae2bf34fdb6505eda6cdb6ea235d1ecf62d21c93c" Oct 31 15:04:40.749856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:40.749837 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg" Oct 31 15:04:41.666215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.666173 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt"] Oct 31 15:04:41.666727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.666713 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4828559a-e3f6-4b7f-8ecd-e006a97b0bba" containerName="revoke-keys" Oct 31 15:04:41.666769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.666728 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4828559a-e3f6-4b7f-8ecd-e006a97b0bba" containerName="revoke-keys" Oct 31 15:04:41.666863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.666853 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4828559a-e3f6-4b7f-8ecd-e006a97b0bba" containerName="revoke-keys" Oct 31 15:04:41.670092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.670071 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.721765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.721732 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7h6\" (UniqueName: \"kubernetes.io/projected/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-kube-api-access-6n7h6\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.721961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.721942 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.722032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.722000 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.726503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.726454 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt"] Oct 31 15:04:41.772273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.772241 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:04:41.772442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.772282 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:04:41.772442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.772326 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:04:41.774263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.774245 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:04:41.822863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.822833 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.822986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.822878 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.822986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.822961 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7h6\" (UniqueName: \"kubernetes.io/projected/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-kube-api-access-6n7h6\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.823621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.823594 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.825313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.825293 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.880797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.880760 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7h6\" (UniqueName: \"kubernetes.io/projected/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-kube-api-access-6n7h6\") pod \"maas-api-revoke-sub-b7510bdf-2x2mt\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:41.985951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:41.985924 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:42.360067 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:42.360038 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dbf432_c3d9_4ca3_80c5_a2e88b6f288a.slice/crio-82db8d0f4adbd14547a7cec208c11fabd1592787a87382b04229cacc3da0f8d1 WatchSource:0}: Error finding container 82db8d0f4adbd14547a7cec208c11fabd1592787a87382b04229cacc3da0f8d1: Status 404 returned error can't find the container with id 82db8d0f4adbd14547a7cec208c11fabd1592787a87382b04229cacc3da0f8d1 Oct 31 15:04:42.390346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:42.390318 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt"] Oct 31 15:04:42.764276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:42.764244 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" event={"ID":"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a","Type":"ContainerStarted","Data":"56f8e0980388c318841a5e7dec5b28b8db210b9c5d37bb0ed76ce0b598e5bc56"} Oct 31 15:04:42.764696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:42.764304 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" event={"ID":"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a","Type":"ContainerStarted","Data":"82db8d0f4adbd14547a7cec208c11fabd1592787a87382b04229cacc3da0f8d1"} Oct 31 15:04:42.890010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:42.889901 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" podStartSLOduration=1.889884222 podStartE2EDuration="1.889884222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:42.887021838 +0000 UTC m=+1128.506573254" watchObservedRunningTime="2026-10-31 15:04:42.889884222 +0000 UTC m=+1128.509435633" Oct 31 15:04:43.773709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:43.773676 4865 generic.go:356] "Generic (PLEG): container finished" podID="a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" containerID="56f8e0980388c318841a5e7dec5b28b8db210b9c5d37bb0ed76ce0b598e5bc56" exitCode=0 Oct 31 15:04:43.774176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:43.773752 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" event={"ID":"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a","Type":"ContainerDied","Data":"56f8e0980388c318841a5e7dec5b28b8db210b9c5d37bb0ed76ce0b598e5bc56"} Oct 31 15:04:44.318153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.318122 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:44.931707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.931682 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:44.997086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.997059 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-xks-ca\") pod \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " Oct 31 15:04:44.997086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.997089 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-openshift-ca\") pod \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " Oct 31 15:04:44.997331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.997132 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-kube-api-access-6n7h6\" (UniqueName: \"kubernetes.io/projected/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-kube-api-access-6n7h6\") pod \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\" (UID: \"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a\") " Oct 31 15:04:44.997536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.997452 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-openshift-ca" pod "a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" (UID: "a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:44.999507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.999464 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-xks-ca" pod "a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" (UID: "a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:44.999635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:44.999564 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-kube-api-access-6n7h6" pod "a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" (UID: "a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a"). InnerVolumeSpecName "kube-api-access-6n7h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:45.021876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.021849 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:45.132358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.100738 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:45.132829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.132718 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:45.132829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.132785 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n7h6\" (UniqueName: \"kubernetes.io/projected/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a-kube-api-access-6n7h6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:45.528380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.528353 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg"] Oct 31 15:04:45.528909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.528895 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" containerName="revoke-keys" Oct 31 15:04:45.528957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.528911 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" containerName="revoke-keys" Oct 31 15:04:45.529027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.529017 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" containerName="revoke-keys" Oct 31 15:04:45.532321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.532305 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.586496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.586457 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg"] Oct 31 15:04:45.638992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.638960 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.639201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.639169 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.639339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.639255 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpl9c\" (UniqueName: \"kubernetes.io/projected/92124fc8-c59a-4d9d-a526-7c31e96804e0-kube-api-access-dpl9c\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.740819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.740782 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.741005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.740921 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.741005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.740964 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpl9c\" (UniqueName: \"kubernetes.io/projected/92124fc8-c59a-4d9d-a526-7c31e96804e0-kube-api-access-dpl9c\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.741617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.741546 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.743333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.743309 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.790156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.790082 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" Oct 31 15:04:45.790156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.790127 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt" event={"ID":"a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a","Type":"ContainerDied","Data":"82db8d0f4adbd14547a7cec208c11fabd1592787a87382b04229cacc3da0f8d1"} Oct 31 15:04:45.790331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.790194 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82db8d0f4adbd14547a7cec208c11fabd1592787a87382b04229cacc3da0f8d1" Oct 31 15:04:45.800300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.800277 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpl9c\" (UniqueName: \"kubernetes.io/projected/92124fc8-c59a-4d9d-a526-7c31e96804e0-kube-api-access-dpl9c\") pod \"maas-api-revoke-sub-465be251-qs2rg\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:45.861495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:45.857717 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:46.049667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:46.049580 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76"] Oct 31 15:04:46.052824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:46.052803 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5e746c2-pdk76"] Oct 31 15:04:46.293541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:46.293517 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg"] Oct 31 15:04:46.293768 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:46.293746 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92124fc8_c59a_4d9d_a526_7c31e96804e0.slice/crio-7f802c98b9ad565e37e4f1f79994a0c6b642520778a509622da2a9361891a85a WatchSource:0}: Error finding container 7f802c98b9ad565e37e4f1f79994a0c6b642520778a509622da2a9361891a85a: Status 404 returned error can't find the container with id 7f802c98b9ad565e37e4f1f79994a0c6b642520778a509622da2a9361891a85a Oct 31 15:04:46.496431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:46.496399 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95240a0b-f005-4fe3-9bde-e900244db43a" path="/var/lib/kubelet/pods/95240a0b-f005-4fe3-9bde-e900244db43a/volumes" Oct 31 15:04:46.799179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:46.799147 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" event={"ID":"92124fc8-c59a-4d9d-a526-7c31e96804e0","Type":"ContainerStarted","Data":"9c99a75f2c1b036dec3dca2b8b39a595ecf1f839c7fff52492b9b50e39d0ea20"} Oct 31 15:04:46.799348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:46.799210 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" event={"ID":"92124fc8-c59a-4d9d-a526-7c31e96804e0","Type":"ContainerStarted","Data":"7f802c98b9ad565e37e4f1f79994a0c6b642520778a509622da2a9361891a85a"} Oct 31 15:04:46.915379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:46.915275 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" podStartSLOduration=1.9152633030000001 podStartE2EDuration="1.915263303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:46.914162784 +0000 UTC m=+1132.533714195" watchObservedRunningTime="2026-10-31 15:04:46.915263303 +0000 UTC m=+1132.534814713" Oct 31 15:04:47.415686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.415654 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:04:47.489906 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:04:47.489872 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:04:47.546972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.546947 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj"] Oct 31 15:04:47.550878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.550862 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.607298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.607273 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj"] Oct 31 15:04:47.683059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.683027 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.683237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.683075 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.683385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.683368 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmpk\" (UniqueName: \"kubernetes.io/projected/bb81f782-5a08-4d0b-b40d-8e8f46f60472-kube-api-access-bmmpk\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.784353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.784272 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmpk\" (UniqueName: \"kubernetes.io/projected/bb81f782-5a08-4d0b-b40d-8e8f46f60472-kube-api-access-bmmpk\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.784353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.784318 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.784353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.784343 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.784991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.784969 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.786817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.786797 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:47.845084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.843823 4865 generic.go:356] "Generic (PLEG): container finished" podID="92124fc8-c59a-4d9d-a526-7c31e96804e0" containerID="9c99a75f2c1b036dec3dca2b8b39a595ecf1f839c7fff52492b9b50e39d0ea20" exitCode=0 Oct 31 15:04:47.845084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.844088 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" event={"ID":"92124fc8-c59a-4d9d-a526-7c31e96804e0","Type":"ContainerDied","Data":"9c99a75f2c1b036dec3dca2b8b39a595ecf1f839c7fff52492b9b50e39d0ea20"} Oct 31 15:04:47.895505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:47.880301 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmpk\" (UniqueName: \"kubernetes.io/projected/bb81f782-5a08-4d0b-b40d-8e8f46f60472-kube-api-access-bmmpk\") pod \"maas-api-revoke-sub-5957f5ba-nqpqj\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:48.164919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:48.164822 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:49.351614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:49.351518 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj"] Oct 31 15:04:49.844497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:49.843774 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:49.967996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:49.966620 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" event={"ID":"bb81f782-5a08-4d0b-b40d-8e8f46f60472","Type":"ContainerStarted","Data":"b28a96815079793bc3e9086282d957d4fe0da38a866542d82148f8718866e98f"} Oct 31 15:04:50.425590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.409804 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:50.441886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.440115 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:50.441886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.440391 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Oct 31 15:04:50.627626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.627583 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:04:50.648523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.647652 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 31 15:04:50.697195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.697101 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:50.716745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.716714 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-xks-ca\") pod \"92124fc8-c59a-4d9d-a526-7c31e96804e0\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " Oct 31 15:04:50.746961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.746784 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92124fc8-c59a-4d9d-a526-7c31e96804e0-kube-api-access-dpl9c\" (UniqueName: \"kubernetes.io/projected/92124fc8-c59a-4d9d-a526-7c31e96804e0-kube-api-access-dpl9c\") pod \"92124fc8-c59a-4d9d-a526-7c31e96804e0\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " Oct 31 15:04:50.746961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.746943 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-openshift-ca\") pod \"92124fc8-c59a-4d9d-a526-7c31e96804e0\" (UID: \"92124fc8-c59a-4d9d-a526-7c31e96804e0\") " Oct 31 15:04:50.791633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.791574 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-xks-ca" pod "92124fc8-c59a-4d9d-a526-7c31e96804e0" (UID: "92124fc8-c59a-4d9d-a526-7c31e96804e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:50.792326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.792295 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92124fc8-c59a-4d9d-a526-7c31e96804e0-kube-api-access-dpl9c" pod "92124fc8-c59a-4d9d-a526-7c31e96804e0" (UID: "92124fc8-c59a-4d9d-a526-7c31e96804e0"). InnerVolumeSpecName "kube-api-access-dpl9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:50.809505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.808562 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:04:50.815507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.813566 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:04:50.819493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.817711 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-openshift-ca" pod "92124fc8-c59a-4d9d-a526-7c31e96804e0" (UID: "92124fc8-c59a-4d9d-a526-7c31e96804e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:50.845501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.845369 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:50.857999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.845771 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:50.857999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.846858 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:04:50.870280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.870243 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:50.870504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.870461 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpl9c\" (UniqueName: \"kubernetes.io/projected/92124fc8-c59a-4d9d-a526-7c31e96804e0-kube-api-access-dpl9c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:50.870645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.870631 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92124fc8-c59a-4d9d-a526-7c31e96804e0-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:50.954569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:50.954447 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:51.247360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.247323 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:51.251445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.249897 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" event={"ID":"bb81f782-5a08-4d0b-b40d-8e8f46f60472","Type":"ContainerStarted","Data":"37d0664dc71b23f4f1dba8074d70a577e00f09d0949be0e5345c97137fcd859a"} Oct 31 15:04:51.272248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.272213 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" event={"ID":"92124fc8-c59a-4d9d-a526-7c31e96804e0","Type":"ContainerDied","Data":"7f802c98b9ad565e37e4f1f79994a0c6b642520778a509622da2a9361891a85a"} Oct 31 15:04:51.301618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.301580 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f802c98b9ad565e37e4f1f79994a0c6b642520778a509622da2a9361891a85a" Oct 31 15:04:51.301868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.272501 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg" Oct 31 15:04:51.353202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.353157 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:04:51.455350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.447420 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:51.513568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.506247 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:04:51.513568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.507675 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:04:51.532258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.532196 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" podStartSLOduration=4.532174465 podStartE2EDuration="4.532174465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:51.38511266 +0000 UTC m=+1137.004664083" watchObservedRunningTime="2026-10-31 15:04:51.532174465 +0000 UTC m=+1137.151725877" Oct 31 15:04:51.653588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.633149 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:04:51.653588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.634260 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:04:51.755497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.732305 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:51.822513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.822404 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:04:51.823792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.823767 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:04:51.962533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.823951 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:04:51.962533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.857857 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:04:51.962950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.962928 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:04:51.964592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.964571 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:04:51.964782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:51.964766 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:04:52.205592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.188996 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:52.211527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.191583 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:52.356494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.355719 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:52.356494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.356325 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:04:52.605619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.605570 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:04:52.788030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.787294 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:52.845432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.842691 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:52.934812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.934502 4865 generic.go:356] "Generic (PLEG): container finished" podID="bb81f782-5a08-4d0b-b40d-8e8f46f60472" containerID="37d0664dc71b23f4f1dba8074d70a577e00f09d0949be0e5345c97137fcd859a" exitCode=0 Oct 31 15:04:52.939159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.939128 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" event={"ID":"bb81f782-5a08-4d0b-b40d-8e8f46f60472","Type":"ContainerDied","Data":"37d0664dc71b23f4f1dba8074d70a577e00f09d0949be0e5345c97137fcd859a"} Oct 31 15:04:52.940356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.939448 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:04:52.940654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.940637 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:04:52.940796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.940784 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:04:52.940924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.940913 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:04:52.941617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.941596 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:04:52.941715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.941644 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:04:52.949194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:52.949164 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:04:54.071767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.071664 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:54.216599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.216566 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:54.294093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.294067 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:54.489162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.486224 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-xks-ca\") pod \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " Oct 31 15:04:54.489162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.486333 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb81f782-5a08-4d0b-b40d-8e8f46f60472-kube-api-access-bmmpk\" (UniqueName: \"kubernetes.io/projected/bb81f782-5a08-4d0b-b40d-8e8f46f60472-kube-api-access-bmmpk\") pod \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " Oct 31 15:04:54.489162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.486397 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-openshift-ca\") pod \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\" (UID: \"bb81f782-5a08-4d0b-b40d-8e8f46f60472\") " Oct 31 15:04:54.489162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.487386 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-openshift-ca" pod "bb81f782-5a08-4d0b-b40d-8e8f46f60472" (UID: "bb81f782-5a08-4d0b-b40d-8e8f46f60472"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:54.489162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.488142 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:54.505285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.504052 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-xks-ca" pod "bb81f782-5a08-4d0b-b40d-8e8f46f60472" (UID: "bb81f782-5a08-4d0b-b40d-8e8f46f60472"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:54.514406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.508184 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb81f782-5a08-4d0b-b40d-8e8f46f60472-kube-api-access-bmmpk" pod "bb81f782-5a08-4d0b-b40d-8e8f46f60472" (UID: "bb81f782-5a08-4d0b-b40d-8e8f46f60472"). InnerVolumeSpecName "kube-api-access-bmmpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:54.589211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.589186 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb81f782-5a08-4d0b-b40d-8e8f46f60472-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:54.589211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:54.589214 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmpk\" (UniqueName: \"kubernetes.io/projected/bb81f782-5a08-4d0b-b40d-8e8f46f60472-kube-api-access-bmmpk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:55.055109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.055080 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" Oct 31 15:04:55.055109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.055097 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj" event={"ID":"bb81f782-5a08-4d0b-b40d-8e8f46f60472","Type":"ContainerDied","Data":"b28a96815079793bc3e9086282d957d4fe0da38a866542d82148f8718866e98f"} Oct 31 15:04:55.055314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.055148 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28a96815079793bc3e9086282d957d4fe0da38a866542d82148f8718866e98f" Oct 31 15:04:55.121384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.121359 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq"] Oct 31 15:04:55.121921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.121907 4865 cpu_manager.go:414] "containerMap: removing container" podUID="92124fc8-c59a-4d9d-a526-7c31e96804e0" containerName="revoke-keys" Oct 31 15:04:55.121960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.121923 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92124fc8-c59a-4d9d-a526-7c31e96804e0" containerName="revoke-keys" Oct 31 15:04:55.121960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.121937 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bb81f782-5a08-4d0b-b40d-8e8f46f60472" containerName="revoke-keys" Oct 31 15:04:55.121960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.121941 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb81f782-5a08-4d0b-b40d-8e8f46f60472" containerName="revoke-keys" Oct 31 15:04:55.122045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.122018 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="92124fc8-c59a-4d9d-a526-7c31e96804e0" containerName="revoke-keys" Oct 31 15:04:55.122045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.122027 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb81f782-5a08-4d0b-b40d-8e8f46f60472" containerName="revoke-keys" Oct 31 15:04:55.126796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.126779 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.181527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.181491 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq"] Oct 31 15:04:55.299594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.299565 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-provider-credentials\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.299741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.299605 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-plugins-config-volume\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.299807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.299736 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-routing-overlay-volume\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.299888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.299873 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snw8\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-kube-api-access-8snw8\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.393702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.393593 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv"] Oct 31 15:04:55.399803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.399778 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.400620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.400596 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snw8\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-kube-api-access-8snw8\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.400736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.400694 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-provider-credentials\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.400736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.400729 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-plugins-config-volume\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.400840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.400812 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-routing-overlay-volume\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.401561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.401538 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-plugins-config-volume\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.401670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.401555 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-routing-overlay-volume\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.403286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.403268 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-provider-credentials\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.449264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.449242 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv"] Oct 31 15:04:55.453315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.453292 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:04:55.453434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.453418 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:04:55.453546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.453497 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:04:55.453546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.453504 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:04:55.458662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.458640 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snw8\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-kube-api-access-8snw8\") pod \"payload-processing-external-model-7c54d48db8-2dzxq\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:55.502205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.502160 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.502321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.502280 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.502366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.502334 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmsx\" (UniqueName: \"kubernetes.io/projected/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-kube-api-access-jxmsx\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.603966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.603911 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.604172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.604019 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmsx\" (UniqueName: \"kubernetes.io/projected/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-kube-api-access-jxmsx\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.604172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.604159 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.604671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.604643 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.606673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.606656 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.662151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.662080 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmsx\" (UniqueName: \"kubernetes.io/projected/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-kube-api-access-jxmsx\") pod \"maas-api-revoke-sub-636fc0bc-tfrrv\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.724154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.724125 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:55.740295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:55.740273 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:56.044498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.033220 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:04:56.195494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.192084 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv"] Oct 31 15:04:56.230075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.230046 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq"] Oct 31 15:04:56.230673 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:56.230603 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92a0b02_b9b4_41c9_9adf_4aa3e11a4da6.slice/crio-8c8d8d7943a9f9fc6ceaf35d7ec27ebaec8ef90393ebf740592aa136c5d65095 WatchSource:0}: Error finding container 8c8d8d7943a9f9fc6ceaf35d7ec27ebaec8ef90393ebf740592aa136c5d65095: Status 404 returned error can't find the container with id 8c8d8d7943a9f9fc6ceaf35d7ec27ebaec8ef90393ebf740592aa136c5d65095 Oct 31 15:04:56.260509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.260486 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd"] Oct 31 15:04:56.265897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.265879 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.321228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.321155 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd"] Oct 31 15:04:56.421109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.421077 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgs2c\" (UniqueName: \"kubernetes.io/projected/84cb418e-35ed-4a22-b50b-542a8aeb011c-kube-api-access-xgs2c\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.421238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.421125 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.421302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.421234 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.522879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.522853 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgs2c\" (UniqueName: \"kubernetes.io/projected/84cb418e-35ed-4a22-b50b-542a8aeb011c-kube-api-access-xgs2c\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.523017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.522889 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.523017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.522931 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.523543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.523520 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.525318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.525300 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.580426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.580363 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgs2c\" (UniqueName: \"kubernetes.io/projected/84cb418e-35ed-4a22-b50b-542a8aeb011c-kube-api-access-xgs2c\") pod \"maas-api-revoke-sub-cdcf77c9-8jzgd\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.582239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.582219 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:56.958769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:56.958736 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd"] Oct 31 15:04:56.961631 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:04:56.961603 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cb418e_35ed_4a22_b50b_542a8aeb011c.slice/crio-d45e8da465f349d5e13d348a162248483a98c1c0bf5cd84a503534ea9cb4efc8 WatchSource:0}: Error finding container d45e8da465f349d5e13d348a162248483a98c1c0bf5cd84a503534ea9cb4efc8: Status 404 returned error can't find the container with id d45e8da465f349d5e13d348a162248483a98c1c0bf5cd84a503534ea9cb4efc8 Oct 31 15:04:57.074326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.074295 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" event={"ID":"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273","Type":"ContainerStarted","Data":"948245c2c71fee8a82f3f9e483a62a36051187787c63a93367bb9da4081ffd4b"} Oct 31 15:04:57.074503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.074354 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" event={"ID":"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273","Type":"ContainerStarted","Data":"b806b76b4514144443b1c246e4e960da153f9ade4e8a800c5e83d3a7bc37175e"} Oct 31 15:04:57.075949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.075928 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" event={"ID":"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6","Type":"ContainerStarted","Data":"173b21e40e9f4bf9116dcb41aed92a5f70c2c1d4772d65fa908b7d4ff8f86862"} Oct 31 15:04:57.076066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.075967 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" event={"ID":"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6","Type":"ContainerStarted","Data":"8c8d8d7943a9f9fc6ceaf35d7ec27ebaec8ef90393ebf740592aa136c5d65095"} Oct 31 15:04:57.076066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.076026 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:04:57.077267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.077248 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" event={"ID":"84cb418e-35ed-4a22-b50b-542a8aeb011c","Type":"ContainerStarted","Data":"d45e8da465f349d5e13d348a162248483a98c1c0bf5cd84a503534ea9cb4efc8"} Oct 31 15:04:57.321986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.321897 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" podStartSLOduration=2.321883017 podStartE2EDuration="2.321883017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:57.190525689 +0000 UTC m=+1142.810077102" watchObservedRunningTime="2026-10-31 15:04:57.321883017 +0000 UTC m=+1142.941434472" Oct 31 15:04:57.323120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:57.323090 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" podStartSLOduration=2.32308037 podStartE2EDuration="2.32308037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:57.319284044 +0000 UTC m=+1142.938835454" watchObservedRunningTime="2026-10-31 15:04:57.32308037 +0000 UTC m=+1142.942631780" Oct 31 15:04:58.085279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:58.085244 4865 generic.go:356] "Generic (PLEG): container finished" podID="84cb418e-35ed-4a22-b50b-542a8aeb011c" containerID="a1ce8b8188bb169e9e373d4bb499c1e63e9b4065a481ff5fc9b82fc0634f80be" exitCode=0 Oct 31 15:04:58.085438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:58.085332 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" event={"ID":"84cb418e-35ed-4a22-b50b-542a8aeb011c","Type":"ContainerDied","Data":"a1ce8b8188bb169e9e373d4bb499c1e63e9b4065a481ff5fc9b82fc0634f80be"} Oct 31 15:04:58.087009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:58.086983 4865 generic.go:356] "Generic (PLEG): container finished" podID="1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" containerID="948245c2c71fee8a82f3f9e483a62a36051187787c63a93367bb9da4081ffd4b" exitCode=0 Oct 31 15:04:58.087090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:58.087066 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" event={"ID":"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273","Type":"ContainerDied","Data":"948245c2c71fee8a82f3f9e483a62a36051187787c63a93367bb9da4081ffd4b"} Oct 31 15:04:59.256705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.256603 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:04:59.291101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.291080 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:04:59.354368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354295 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-openshift-ca\") pod \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " Oct 31 15:04:59.354368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354336 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-kube-api-access-jxmsx\" (UniqueName: \"kubernetes.io/projected/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-kube-api-access-jxmsx\") pod \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " Oct 31 15:04:59.354588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354392 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-xks-ca\") pod \"84cb418e-35ed-4a22-b50b-542a8aeb011c\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " Oct 31 15:04:59.354588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354452 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84cb418e-35ed-4a22-b50b-542a8aeb011c-kube-api-access-xgs2c\" (UniqueName: \"kubernetes.io/projected/84cb418e-35ed-4a22-b50b-542a8aeb011c-kube-api-access-xgs2c\") pod \"84cb418e-35ed-4a22-b50b-542a8aeb011c\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " Oct 31 15:04:59.354588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354527 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-openshift-ca\") pod \"84cb418e-35ed-4a22-b50b-542a8aeb011c\" (UID: \"84cb418e-35ed-4a22-b50b-542a8aeb011c\") " Oct 31 15:04:59.354751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354651 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-openshift-ca" pod "1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" (UID: "1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:59.354751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354721 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-xks-ca\") pod \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\" (UID: \"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273\") " Oct 31 15:04:59.354909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.354889 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-openshift-ca" pod "84cb418e-35ed-4a22-b50b-542a8aeb011c" (UID: "84cb418e-35ed-4a22-b50b-542a8aeb011c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:04:59.355234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.355215 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:59.355314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.355238 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:59.356831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.356810 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-xks-ca" pod "84cb418e-35ed-4a22-b50b-542a8aeb011c" (UID: "84cb418e-35ed-4a22-b50b-542a8aeb011c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:59.356944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.356933 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-xks-ca" pod "1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" (UID: "1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:04:59.357186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.357158 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cb418e-35ed-4a22-b50b-542a8aeb011c-kube-api-access-xgs2c" pod "84cb418e-35ed-4a22-b50b-542a8aeb011c" (UID: "84cb418e-35ed-4a22-b50b-542a8aeb011c"). InnerVolumeSpecName "kube-api-access-xgs2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:59.357283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.357180 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-kube-api-access-jxmsx" pod "1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" (UID: "1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273"). InnerVolumeSpecName "kube-api-access-jxmsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:04:59.456863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.456831 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb418e-35ed-4a22-b50b-542a8aeb011c-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:59.456863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.456859 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgs2c\" (UniqueName: \"kubernetes.io/projected/84cb418e-35ed-4a22-b50b-542a8aeb011c-kube-api-access-xgs2c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:59.456863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.456870 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:59.457091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:04:59.456880 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmsx\" (UniqueName: \"kubernetes.io/projected/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273-kube-api-access-jxmsx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:04:59.558446 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:04:59.558418 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-be9e69b601a9: secret "agc-cred-be9e69b601a9" not found Oct 31 15:04:59.558446 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:04:59.558446 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2aa17aef255: secret "agc-cred-e2aa17aef255" not found Oct 31 15:04:59.558724 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:04:59.558505 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc: [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:04:59.558724 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:04:59.558571 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials podName:7815f242-d478-4f5c-9af3-cd642e0a876b nodeName:}" failed. No retries permitted until 2026-10-31 15:05:00.058552005 +0000 UTC m=+1145.678103411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials") pod "payload-processing-external-model-7b5d499d94-ht4zc" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b") : [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:00.063852 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:00.063821 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-be9e69b601a9: secret "agc-cred-be9e69b601a9" not found Oct 31 15:05:00.063852 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:00.063847 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2aa17aef255: secret "agc-cred-e2aa17aef255" not found Oct 31 15:05:00.063852 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:00.063860 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc: [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:00.064165 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:00.063938 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials podName:7815f242-d478-4f5c-9af3-cd642e0a876b nodeName:}" failed. No retries permitted until 2026-10-31 15:05:01.063918719 +0000 UTC m=+1146.683470125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials") pod "payload-processing-external-model-7b5d499d94-ht4zc" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b") : [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:00.102015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:00.101987 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" Oct 31 15:05:00.102157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:00.102025 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd" event={"ID":"84cb418e-35ed-4a22-b50b-542a8aeb011c","Type":"ContainerDied","Data":"d45e8da465f349d5e13d348a162248483a98c1c0bf5cd84a503534ea9cb4efc8"} Oct 31 15:05:00.102157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:00.102074 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d45e8da465f349d5e13d348a162248483a98c1c0bf5cd84a503534ea9cb4efc8" Oct 31 15:05:00.103407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:00.103384 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" event={"ID":"1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273","Type":"ContainerDied","Data":"b806b76b4514144443b1c246e4e960da153f9ade4e8a800c5e83d3a7bc37175e"} Oct 31 15:05:00.103547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:00.103392 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv" Oct 31 15:05:00.103547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:00.103429 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b806b76b4514144443b1c246e4e960da153f9ade4e8a800c5e83d3a7bc37175e" Oct 31 15:05:00.801301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:00.801271 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:01.093607 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:01.093526 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-be9e69b601a9: secret "agc-cred-be9e69b601a9" not found Oct 31 15:05:01.093607 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:01.093552 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2aa17aef255: secret "agc-cred-e2aa17aef255" not found Oct 31 15:05:01.093607 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:01.093561 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc: [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:01.093831 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:01.093613 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials podName:7815f242-d478-4f5c-9af3-cd642e0a876b nodeName:}" failed. No retries permitted until 2026-10-31 15:05:03.09359996 +0000 UTC m=+1148.713151350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials") pod "payload-processing-external-model-7b5d499d94-ht4zc" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b") : [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:03.119427 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:03.119395 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-be9e69b601a9: secret "agc-cred-be9e69b601a9" not found Oct 31 15:05:03.119427 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:03.119425 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2aa17aef255: secret "agc-cred-e2aa17aef255" not found Oct 31 15:05:03.119901 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:03.119436 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc: [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:03.119901 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:03.119518 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials podName:7815f242-d478-4f5c-9af3-cd642e0a876b nodeName:}" failed. No retries permitted until 2026-10-31 15:05:07.119499516 +0000 UTC m=+1152.739050919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials") pod "payload-processing-external-model-7b5d499d94-ht4zc" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b") : [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:05.532665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.532609 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:05.605505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.602994 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d"] Oct 31 15:05:05.605505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.603973 4865 cpu_manager.go:414] "containerMap: removing container" podUID="84cb418e-35ed-4a22-b50b-542a8aeb011c" containerName="revoke-keys" Oct 31 15:05:05.605505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.603993 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84cb418e-35ed-4a22-b50b-542a8aeb011c" containerName="revoke-keys" Oct 31 15:05:05.605505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.604017 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" containerName="revoke-keys" Oct 31 15:05:05.605505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.604025 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" containerName="revoke-keys" Oct 31 15:05:05.605505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.604233 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="84cb418e-35ed-4a22-b50b-542a8aeb011c" containerName="revoke-keys" Oct 31 15:05:05.605505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.604246 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" containerName="revoke-keys" Oct 31 15:05:05.618090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.618059 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.666331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.666293 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d"] Oct 31 15:05:05.698876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.698846 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:05:05.698876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.698846 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:05:05.699083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.698958 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:05:05.722587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.722558 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:05:05.756305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.756274 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.756540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.756448 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n48jp\" (UniqueName: \"kubernetes.io/projected/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-kube-api-access-n48jp\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.756626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.756563 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.857742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.857663 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.857908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.857807 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n48jp\" (UniqueName: \"kubernetes.io/projected/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-kube-api-access-n48jp\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.857908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.857869 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.858383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.858363 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.860405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.860386 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.915699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.915667 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n48jp\" (UniqueName: \"kubernetes.io/projected/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-kube-api-access-n48jp\") pod \"maas-api-revoke-sub-7bf39bc0-xs28d\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:05.962114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:05.962083 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:06.049062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:06.049022 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl"] Oct 31 15:05:06.052197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:06.052176 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808a6b23-hxkwl"] Oct 31 15:05:06.344203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:06.344177 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d"] Oct 31 15:05:06.344593 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:05:06.344572 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf15f47_91e2_400e_9a8e_4444c0e0b9dd.slice/crio-eda4e6e9ae4b1559672aa4e8d751a3211736dc5baf2b4f159c1e3c0ed41fb1c1 WatchSource:0}: Error finding container eda4e6e9ae4b1559672aa4e8d751a3211736dc5baf2b4f159c1e3c0ed41fb1c1: Status 404 returned error can't find the container with id eda4e6e9ae4b1559672aa4e8d751a3211736dc5baf2b4f159c1e3c0ed41fb1c1 Oct 31 15:05:06.498825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:06.498793 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6260853-430c-4504-a744-5ac2f7a36430" path="/var/lib/kubelet/pods/f6260853-430c-4504-a744-5ac2f7a36430/volumes" Oct 31 15:05:07.152171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:07.152145 4865 generic.go:356] "Generic (PLEG): container finished" podID="2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" containerID="38818d7fc8e5523f5ff090514401c15ef48e556c144883d8b9114486532d1576" exitCode=0 Oct 31 15:05:07.152535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:07.152225 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" event={"ID":"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd","Type":"ContainerDied","Data":"38818d7fc8e5523f5ff090514401c15ef48e556c144883d8b9114486532d1576"} Oct 31 15:05:07.152535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:07.152283 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" event={"ID":"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd","Type":"ContainerStarted","Data":"eda4e6e9ae4b1559672aa4e8d751a3211736dc5baf2b4f159c1e3c0ed41fb1c1"} Oct 31 15:05:07.175052 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:07.175032 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-be9e69b601a9: secret "agc-cred-be9e69b601a9" not found Oct 31 15:05:07.175052 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:07.175051 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2aa17aef255: secret "agc-cred-e2aa17aef255" not found Oct 31 15:05:07.175165 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:07.175060 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc: [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:07.175165 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:07.175111 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials podName:7815f242-d478-4f5c-9af3-cd642e0a876b nodeName:}" failed. No retries permitted until 2026-10-31 15:05:15.175097721 +0000 UTC m=+1160.794649110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials") pod "payload-processing-external-model-7b5d499d94-ht4zc" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b") : [secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:07.638793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:07.628654 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:07.776489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:07.776441 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:08.089686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.089653 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" podUID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:05:08.316200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.316176 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:08.391040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.390954 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-kube-api-access-n48jp\" (UniqueName: \"kubernetes.io/projected/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-kube-api-access-n48jp\") pod \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " Oct 31 15:05:08.391206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.391060 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-xks-ca\") pod \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " Oct 31 15:05:08.391206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.391110 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-openshift-ca\") pod \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\" (UID: \"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd\") " Oct 31 15:05:08.391502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.391452 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-openshift-ca" pod "2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" (UID: "2bf15f47-91e2-400e-9a8e-4444c0e0b9dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:08.391683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.391665 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:08.393100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.393078 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-xks-ca" pod "2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" (UID: "2bf15f47-91e2-400e-9a8e-4444c0e0b9dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:05:08.393152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.393139 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-kube-api-access-n48jp" pod "2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" (UID: "2bf15f47-91e2-400e-9a8e-4444c0e0b9dd"). InnerVolumeSpecName "kube-api-access-n48jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:08.492611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.492574 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n48jp\" (UniqueName: \"kubernetes.io/projected/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-kube-api-access-n48jp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:08.492611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:08.492605 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:09.166676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:09.166646 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" Oct 31 15:05:09.166845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:09.166678 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d" event={"ID":"2bf15f47-91e2-400e-9a8e-4444c0e0b9dd","Type":"ContainerDied","Data":"eda4e6e9ae4b1559672aa4e8d751a3211736dc5baf2b4f159c1e3c0ed41fb1c1"} Oct 31 15:05:09.166845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:09.166722 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda4e6e9ae4b1559672aa4e8d751a3211736dc5baf2b4f159c1e3c0ed41fb1c1" Oct 31 15:05:09.276781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:09.276753 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc"] Oct 31 15:05:09.280078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:09.280055 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq"] Oct 31 15:05:09.281276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:09.281254 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" podUID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:05:09.416111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:09.416082 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:10.228497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:10.222152 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:12.229652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.229601 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.519666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.519568 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.713979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.713936 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.714671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.714246 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.718756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.718728 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.782669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.782559 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.791602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.791559 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.845611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:12.845562 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.135550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.134545 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.618616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.618569 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:05:13.660374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.660340 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:05:13.747057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.713844 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.797603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.718532 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:05:13.797603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.718576 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.797603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.759372 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:05:13.797603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.759430 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:05:13.797603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.729608 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.797603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.760565 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:05:13.798260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.798219 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.799268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.799244 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:05:13.839521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.838909 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.839521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.839273 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:05:13.880583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.871214 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.880583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:13.871710 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:05:14.217703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.201459 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:14.217703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.201836 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:05:14.292094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.279541 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" podUID="7815f242-d478-4f5c-9af3-cd642e0a876b" containerName="payload-processing" containerID="cri-o://9fe17b116223ad896dac076994fb6f3477ac57ac4b65e9ea6651ba917443856c" gracePeriod=25 Oct 31 15:05:14.296835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.293603 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" podUID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" containerName="payload-processing" containerID="cri-o://173b21e40e9f4bf9116dcb41aed92a5f70c2c1d4772d65fa908b7d4ff8f86862" gracePeriod=25 Oct 31 15:05:14.677721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.677648 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:14.796557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.772092 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:14.821805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.776636 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:05:14.860314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.776681 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:05:14.860314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.798553 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:14.860314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.854914 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:14.860314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.855103 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 31 15:05:14.947809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:14.947759 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:05:15.201092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.200990 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:15.214602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.212814 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:15.217353 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:15.216688 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-5e686c13c1d4: secret "agc-cred-5e686c13c1d4" not found Oct 31 15:05:15.217353 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:15.216715 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-79e80514dad4: secret "agc-cred-79e80514dad4" not found Oct 31 15:05:15.217353 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:15.216723 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-be9e69b601a9: secret "agc-cred-be9e69b601a9" not found Oct 31 15:05:15.217353 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:15.216730 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2aa17aef255: secret "agc-cred-e2aa17aef255" not found Oct 31 15:05:15.217353 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:15.216745 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc: [secret "agc-cred-5e686c13c1d4" not found, secret "agc-cred-79e80514dad4" not found, secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:15.217353 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:15.216805 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials podName:7815f242-d478-4f5c-9af3-cd642e0a876b nodeName:}" failed. No retries permitted until 2026-10-31 15:05:31.216786085 +0000 UTC m=+1176.836337493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials") pod "payload-processing-external-model-7b5d499d94-ht4zc" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b") : [secret "agc-cred-5e686c13c1d4" not found, secret "agc-cred-79e80514dad4" not found, secret "agc-cred-be9e69b601a9" not found, secret "agc-cred-e2aa17aef255" not found] Oct 31 15:05:15.712994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.712944 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:05:15.821703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.821661 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:05:15.902590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.895088 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:05:15.929504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.925763 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:05:16.010601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.958089 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:05:16.010601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:15.969368 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:05:16.011778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.011737 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:16.045498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.044627 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.045498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.045027 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.275552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.274803 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.353495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.330871 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:16.353495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.330969 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.367495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.364050 4865 generic.go:356] "Generic (PLEG): container finished" podID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" containerID="173b21e40e9f4bf9116dcb41aed92a5f70c2c1d4772d65fa908b7d4ff8f86862" exitCode=0 Oct 31 15:05:16.374492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.368125 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" event={"ID":"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6","Type":"ContainerDied","Data":"173b21e40e9f4bf9116dcb41aed92a5f70c2c1d4772d65fa908b7d4ff8f86862"} Oct 31 15:05:16.478505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.446542 4865 generic.go:356] "Generic (PLEG): container finished" podID="7815f242-d478-4f5c-9af3-cd642e0a876b" containerID="9fe17b116223ad896dac076994fb6f3477ac57ac4b65e9ea6651ba917443856c" exitCode=0 Oct 31 15:05:16.482441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.480777 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" event={"ID":"7815f242-d478-4f5c-9af3-cd642e0a876b","Type":"ContainerDied","Data":"9fe17b116223ad896dac076994fb6f3477ac57ac4b65e9ea6651ba917443856c"} Oct 31 15:05:16.750726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.750673 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.903221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.903122 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.905283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.905248 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.905563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:16.905540 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:05:17.009960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.009189 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:17.010240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.010214 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:17.021586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.021544 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:17.161599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.151895 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:05:17.361504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.357560 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx"] Oct 31 15:05:17.361504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.358393 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" containerName="revoke-keys" Oct 31 15:05:17.361504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.358409 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" containerName="revoke-keys" Oct 31 15:05:17.361504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.358552 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" containerName="revoke-keys" Oct 31 15:05:17.447916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.447813 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:05:17.456492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.454674 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx"] Oct 31 15:05:17.456492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.454792 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.471145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.470867 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:05:17.477582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.477559 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:05:17.497916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.497890 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:05:17.504959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.504937 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:05:17.517914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.517869 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:05:17.518262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.518243 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:05:17.523600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.519211 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:05:17.523832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.520838 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:05:17.534273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.534250 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" Oct 31 15:05:17.536823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.534797 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq" event={"ID":"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6","Type":"ContainerDied","Data":"8c8d8d7943a9f9fc6ceaf35d7ec27ebaec8ef90393ebf740592aa136c5d65095"} Oct 31 15:05:17.538291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.538252 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" event={"ID":"7815f242-d478-4f5c-9af3-cd642e0a876b","Type":"ContainerDied","Data":"2640a3db9fd3178312b2212efc6594f23f0679396008e664c2b6f227b054ecee"} Oct 31 15:05:17.538513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.536482 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc" Oct 31 15:05:17.538737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.538420 4865 scope.go:122] "RemoveContainer" containerID="173b21e40e9f4bf9116dcb41aed92a5f70c2c1d4772d65fa908b7d4ff8f86862" Oct 31 15:05:17.563120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.562761 4865 scope.go:122] "RemoveContainer" containerID="9fe17b116223ad896dac076994fb6f3477ac57ac4b65e9ea6651ba917443856c" Oct 31 15:05:17.637067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.637032 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-routing-overlay-volume\") pod \"7815f242-d478-4f5c-9af3-cd642e0a876b\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " Oct 31 15:05:17.637400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.637374 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials\") pod \"7815f242-d478-4f5c-9af3-cd642e0a876b\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " Oct 31 15:05:17.649499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.637609 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-plugins-config-volume\") pod \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " Oct 31 15:05:17.649760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.649735 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-plugins-config-volume\") pod \"7815f242-d478-4f5c-9af3-cd642e0a876b\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " Oct 31 15:05:17.649942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.649923 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-kube-api-access-mxgcf\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-kube-api-access-mxgcf\") pod \"7815f242-d478-4f5c-9af3-cd642e0a876b\" (UID: \"7815f242-d478-4f5c-9af3-cd642e0a876b\") " Oct 31 15:05:17.650080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.650064 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-provider-credentials\") pod \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " Oct 31 15:05:17.650233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.650215 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-routing-overlay-volume\") pod \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " Oct 31 15:05:17.650364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.650349 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-kube-api-access-8snw8\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-kube-api-access-8snw8\") pod \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\" (UID: \"c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6\") " Oct 31 15:05:17.650787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.650760 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.650973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.650954 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smhq\" (UniqueName: \"kubernetes.io/projected/3bb2c10f-aaa7-4679-bd27-3410940c2202-kube-api-access-5smhq\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.651457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.651435 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.651720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.638878 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-routing-overlay-volume" pod "7815f242-d478-4f5c-9af3-cd642e0a876b" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:17.725553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.641717 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-plugins-config-volume" pod "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" (UID: "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:17.770500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.763405 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.770500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.763511 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smhq\" (UniqueName: \"kubernetes.io/projected/3bb2c10f-aaa7-4679-bd27-3410940c2202-kube-api-access-5smhq\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.770500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.763874 4865 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-routing-overlay-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:17.770500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.763899 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:17.770500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.764778 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-routing-overlay-volume" pod "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" (UID: "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:17.770500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.765744 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-plugins-config-volume" pod "7815f242-d478-4f5c-9af3-cd642e0a876b" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:17.798005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.790736 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.816564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.816509 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-kube-api-access-8snw8" pod "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" (UID: "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6"). InnerVolumeSpecName "kube-api-access-8snw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:17.816915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.816873 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials" pod "7815f242-d478-4f5c-9af3-cd642e0a876b" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:17.857847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.826428 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:05:17.868141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.867431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.868141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.867617 4865 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-routing-overlay-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:17.868141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.867637 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8snw8\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-kube-api-access-8snw8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:17.868141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.867653 4865 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-provider-credentials\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:17.868141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.867668 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7815f242-d478-4f5c-9af3-cd642e0a876b-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:17.884491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.871333 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smhq\" (UniqueName: \"kubernetes.io/projected/3bb2c10f-aaa7-4679-bd27-3410940c2202-kube-api-access-5smhq\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:17.924809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:17.918078 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.046606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.018232 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.083497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.027419 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.152495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.134671 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-kube-api-access-mxgcf" pod "7815f242-d478-4f5c-9af3-cd642e0a876b" (UID: "7815f242-d478-4f5c-9af3-cd642e0a876b"). InnerVolumeSpecName "kube-api-access-mxgcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:18.152495 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:18.134795 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:05:18.249987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.227654 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxgcf\" (UniqueName: \"kubernetes.io/projected/7815f242-d478-4f5c-9af3-cd642e0a876b-kube-api-access-mxgcf\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:18.249987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.227704 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:05:18.278509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.273747 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52965934-sw4lx\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:18.397843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.342933 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:05:18.436855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.436821 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:18.450103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.448577 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.509158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.509102 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-provider-credentials" pod "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" (UID: "c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:18.509773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.509750 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc"] Oct 31 15:05:18.528417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.527797 4865 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6-provider-credentials\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:18.809517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.802053 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:05:18.809517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.802127 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:05:18.809517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.802149 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b5d499d94-ht4zc"] Oct 31 15:05:18.869858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.869833 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:05:18.887605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.887564 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:18.887864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.887839 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.927569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:18.927530 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:05:19.012329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:19.012287 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:05:19.348276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:19.348239 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq"] Oct 31 15:05:19.348663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:19.348647 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-2dzxq"] Oct 31 15:05:19.775778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:19.772535 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx"] Oct 31 15:05:19.864679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:19.864633 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" event={"ID":"3bb2c10f-aaa7-4679-bd27-3410940c2202","Type":"ContainerStarted","Data":"40254002c2d8c47af2c7d0e897d8aa5539f821d80fe2006faa8b6dc9421ebc64"} Oct 31 15:05:20.180569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:20.180528 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:20.498695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:20.498665 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7815f242-d478-4f5c-9af3-cd642e0a876b" path="/var/lib/kubelet/pods/7815f242-d478-4f5c-9af3-cd642e0a876b/volumes" Oct 31 15:05:20.499248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:20.499230 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" path="/var/lib/kubelet/pods/c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6/volumes" Oct 31 15:05:20.872974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:20.872884 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" event={"ID":"3bb2c10f-aaa7-4679-bd27-3410940c2202","Type":"ContainerStarted","Data":"7d2718651e568fc59a5ab94cbc5e2f564ac1f59f0b7503b861632797cdacbd26"} Oct 31 15:05:20.991555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:20.991506 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" podStartSLOduration=4.991489297 podStartE2EDuration="4.991489297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:20.98783551 +0000 UTC m=+1166.607386921" watchObservedRunningTime="2026-10-31 15:05:20.991489297 +0000 UTC m=+1166.611040708" Oct 31 15:05:21.429178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.429146 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp"] Oct 31 15:05:21.429715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.429700 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" containerName="payload-processing" Oct 31 15:05:21.429787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.429716 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" containerName="payload-processing" Oct 31 15:05:21.429787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.429751 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7815f242-d478-4f5c-9af3-cd642e0a876b" containerName="payload-processing" Oct 31 15:05:21.429787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.429757 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7815f242-d478-4f5c-9af3-cd642e0a876b" containerName="payload-processing" Oct 31 15:05:21.429880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.429842 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7815f242-d478-4f5c-9af3-cd642e0a876b" containerName="payload-processing" Oct 31 15:05:21.429880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.429852 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92a0b02-b9b4-41c9-9adf-4aa3e11a4da6" containerName="payload-processing" Oct 31 15:05:21.433253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.433237 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.489579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.489550 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp"] Oct 31 15:05:21.557502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.557446 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.557696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.557597 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jt4k\" (UniqueName: \"kubernetes.io/projected/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-kube-api-access-9jt4k\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.557851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.557830 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.659553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.659520 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.659710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.659627 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.659710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.659690 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jt4k\" (UniqueName: \"kubernetes.io/projected/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-kube-api-access-9jt4k\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.665177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.660861 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.670209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.670171 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.718625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.718600 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jt4k\" (UniqueName: \"kubernetes.io/projected/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-kube-api-access-9jt4k\") pod \"maas-api-revoke-sub-b30b9277-8z4sp\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.746398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.746375 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:21.883497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.883427 4865 generic.go:356] "Generic (PLEG): container finished" podID="3bb2c10f-aaa7-4679-bd27-3410940c2202" containerID="7d2718651e568fc59a5ab94cbc5e2f564ac1f59f0b7503b861632797cdacbd26" exitCode=0 Oct 31 15:05:21.884007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:21.883574 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" event={"ID":"3bb2c10f-aaa7-4679-bd27-3410940c2202","Type":"ContainerDied","Data":"7d2718651e568fc59a5ab94cbc5e2f564ac1f59f0b7503b861632797cdacbd26"} Oct 31 15:05:22.124098 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:05:22.124067 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306ad577_b94e_45ca_98a1_ed3eaf7af4ed.slice/crio-84b6fda6d52b0da5b4bf2fd5df07139e2422a9c92ba729510c8c4e5710cefd8a WatchSource:0}: Error finding container 84b6fda6d52b0da5b4bf2fd5df07139e2422a9c92ba729510c8c4e5710cefd8a: Status 404 returned error can't find the container with id 84b6fda6d52b0da5b4bf2fd5df07139e2422a9c92ba729510c8c4e5710cefd8a Oct 31 15:05:22.124242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:22.124132 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp"] Oct 31 15:05:22.891671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:22.891636 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" event={"ID":"306ad577-b94e-45ca-98a1-ed3eaf7af4ed","Type":"ContainerStarted","Data":"93d74ae81e97d67fe2f3e25d98aa8809f738d182dae603e68eb149257779a762"} Oct 31 15:05:22.892143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:22.891688 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" event={"ID":"306ad577-b94e-45ca-98a1-ed3eaf7af4ed","Type":"ContainerStarted","Data":"84b6fda6d52b0da5b4bf2fd5df07139e2422a9c92ba729510c8c4e5710cefd8a"} Oct 31 15:05:23.008345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.008288 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" podStartSLOduration=2.008274468 podStartE2EDuration="2.008274468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:23.006488163 +0000 UTC m=+1168.626039571" watchObservedRunningTime="2026-10-31 15:05:23.008274468 +0000 UTC m=+1168.627825879" Oct 31 15:05:23.045147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.045124 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:23.178612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.178519 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb2c10f-aaa7-4679-bd27-3410940c2202-kube-api-access-5smhq\" (UniqueName: \"kubernetes.io/projected/3bb2c10f-aaa7-4679-bd27-3410940c2202-kube-api-access-5smhq\") pod \"3bb2c10f-aaa7-4679-bd27-3410940c2202\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " Oct 31 15:05:23.178799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.178662 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-openshift-ca\") pod \"3bb2c10f-aaa7-4679-bd27-3410940c2202\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " Oct 31 15:05:23.178799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.178711 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-xks-ca\") pod \"3bb2c10f-aaa7-4679-bd27-3410940c2202\" (UID: \"3bb2c10f-aaa7-4679-bd27-3410940c2202\") " Oct 31 15:05:23.179024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.178999 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-openshift-ca" pod "3bb2c10f-aaa7-4679-bd27-3410940c2202" (UID: "3bb2c10f-aaa7-4679-bd27-3410940c2202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:23.179197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.179180 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:23.180923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.180896 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-xks-ca" pod "3bb2c10f-aaa7-4679-bd27-3410940c2202" (UID: "3bb2c10f-aaa7-4679-bd27-3410940c2202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:05:23.181330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.181305 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb2c10f-aaa7-4679-bd27-3410940c2202-kube-api-access-5smhq" pod "3bb2c10f-aaa7-4679-bd27-3410940c2202" (UID: "3bb2c10f-aaa7-4679-bd27-3410940c2202"). InnerVolumeSpecName "kube-api-access-5smhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:23.280958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.280925 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smhq\" (UniqueName: \"kubernetes.io/projected/3bb2c10f-aaa7-4679-bd27-3410940c2202-kube-api-access-5smhq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:23.280958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.280958 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb2c10f-aaa7-4679-bd27-3410940c2202-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:23.899612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.899581 4865 generic.go:356] "Generic (PLEG): container finished" podID="306ad577-b94e-45ca-98a1-ed3eaf7af4ed" containerID="93d74ae81e97d67fe2f3e25d98aa8809f738d182dae603e68eb149257779a762" exitCode=0 Oct 31 15:05:23.900066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.899669 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" event={"ID":"306ad577-b94e-45ca-98a1-ed3eaf7af4ed","Type":"ContainerDied","Data":"93d74ae81e97d67fe2f3e25d98aa8809f738d182dae603e68eb149257779a762"} Oct 31 15:05:23.901032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.901007 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" Oct 31 15:05:23.901141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.901010 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx" event={"ID":"3bb2c10f-aaa7-4679-bd27-3410940c2202","Type":"ContainerDied","Data":"40254002c2d8c47af2c7d0e897d8aa5539f821d80fe2006faa8b6dc9421ebc64"} Oct 31 15:05:23.901141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:23.901131 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40254002c2d8c47af2c7d0e897d8aa5539f821d80fe2006faa8b6dc9421ebc64" Oct 31 15:05:25.069837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.069812 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:25.205110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.205019 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-xks-ca\") pod \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " Oct 31 15:05:25.205279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.205261 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-kube-api-access-9jt4k\" (UniqueName: \"kubernetes.io/projected/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-kube-api-access-9jt4k\") pod \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " Oct 31 15:05:25.205376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.205352 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-openshift-ca\") pod \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\" (UID: \"306ad577-b94e-45ca-98a1-ed3eaf7af4ed\") " Oct 31 15:05:25.205962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.205869 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-openshift-ca" pod "306ad577-b94e-45ca-98a1-ed3eaf7af4ed" (UID: "306ad577-b94e-45ca-98a1-ed3eaf7af4ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:25.206181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.206156 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:25.207874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.207846 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-xks-ca" pod "306ad577-b94e-45ca-98a1-ed3eaf7af4ed" (UID: "306ad577-b94e-45ca-98a1-ed3eaf7af4ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:05:25.208297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.208270 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-kube-api-access-9jt4k" pod "306ad577-b94e-45ca-98a1-ed3eaf7af4ed" (UID: "306ad577-b94e-45ca-98a1-ed3eaf7af4ed"). InnerVolumeSpecName "kube-api-access-9jt4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:25.262575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.262543 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:25.307787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.307757 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:25.307787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.307786 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jt4k\" (UniqueName: \"kubernetes.io/projected/306ad577-b94e-45ca-98a1-ed3eaf7af4ed-kube-api-access-9jt4k\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:25.916314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.916279 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" event={"ID":"306ad577-b94e-45ca-98a1-ed3eaf7af4ed","Type":"ContainerDied","Data":"84b6fda6d52b0da5b4bf2fd5df07139e2422a9c92ba729510c8c4e5710cefd8a"} Oct 31 15:05:25.916508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.916337 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84b6fda6d52b0da5b4bf2fd5df07139e2422a9c92ba729510c8c4e5710cefd8a" Oct 31 15:05:25.916508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:25.916291 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp" Oct 31 15:05:26.049445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:26.049411 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc"] Oct 31 15:05:26.052741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:26.052719 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2552ae1-pj5dc"] Oct 31 15:05:26.496741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:26.496708 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd908f19-bab0-4fba-ad9d-060798e799b7" path="/var/lib/kubelet/pods/bd908f19-bab0-4fba-ad9d-060798e799b7/volumes" Oct 31 15:05:27.044345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.044311 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q"] Oct 31 15:05:27.044860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.044846 4865 cpu_manager.go:414] "containerMap: removing container" podUID="306ad577-b94e-45ca-98a1-ed3eaf7af4ed" containerName="revoke-keys" Oct 31 15:05:27.044924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.044863 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306ad577-b94e-45ca-98a1-ed3eaf7af4ed" containerName="revoke-keys" Oct 31 15:05:27.044924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.044881 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3bb2c10f-aaa7-4679-bd27-3410940c2202" containerName="revoke-keys" Oct 31 15:05:27.044924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.044886 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb2c10f-aaa7-4679-bd27-3410940c2202" containerName="revoke-keys" Oct 31 15:05:27.045018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.044998 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="306ad577-b94e-45ca-98a1-ed3eaf7af4ed" containerName="revoke-keys" Oct 31 15:05:27.045018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.045009 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb2c10f-aaa7-4679-bd27-3410940c2202" containerName="revoke-keys" Oct 31 15:05:27.048694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.048678 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129298 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129345 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129452 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129548 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129589 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129655 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129688 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef24796-a595-498c-bbbc-7b6fabdccc49-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129761 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89999\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-kube-api-access-89999\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.129819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.129795 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.157569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.157542 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q"] Oct 31 15:05:27.230827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.230802 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.230842 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.230875 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.230904 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.230935 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef24796-a595-498c-bbbc-7b6fabdccc49-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231008 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89999\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-kube-api-access-89999\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231049 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231140 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231192 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231253 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231305 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231339 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.231551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.231536 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.232040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.232019 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef24796-a595-498c-bbbc-7b6fabdccc49-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.233647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.233619 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.233803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.233786 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.289275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.289235 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.289456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.289321 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89999\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-kube-api-access-89999\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mrc8q\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.362515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.362410 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:27.788813 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:05:27.788777 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef24796_a595_498c_bbbc_7b6fabdccc49.slice/crio-996002e856f7545fb66d4f161c3e047694589e8c499b3b0902723c45a36d3dfc WatchSource:0}: Error finding container 996002e856f7545fb66d4f161c3e047694589e8c499b3b0902723c45a36d3dfc: Status 404 returned error can't find the container with id 996002e856f7545fb66d4f161c3e047694589e8c499b3b0902723c45a36d3dfc Oct 31 15:05:27.789503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.789452 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q"] Oct 31 15:05:27.791041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.791009 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:05:27.791130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.791070 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:05:27.791130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.791103 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:05:27.931978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.931937 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" event={"ID":"0ef24796-a595-498c-bbbc-7b6fabdccc49","Type":"ContainerStarted","Data":"e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8"} Oct 31 15:05:27.932302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:27.932039 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" event={"ID":"0ef24796-a595-498c-bbbc-7b6fabdccc49","Type":"ContainerStarted","Data":"996002e856f7545fb66d4f161c3e047694589e8c499b3b0902723c45a36d3dfc"} Oct 31 15:05:28.052782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:28.052679 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podStartSLOduration=2.052664648 podStartE2EDuration="2.052664648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:28.049612569 +0000 UTC m=+1173.669163979" watchObservedRunningTime="2026-10-31 15:05:28.052664648 +0000 UTC m=+1173.672216058" Oct 31 15:05:28.363669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:28.363596 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:28.365292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:28.365266 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 15:05:28.365440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:28.365323 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 15:05:29.363027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:29.362990 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 15:05:29.363447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:29.363057 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 15:05:30.019953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:30.019918 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:30.363302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:30.363214 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 15:05:30.363302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:30.363268 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 15:05:31.363770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:31.363736 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 15:05:31.364190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:31.363800 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 15:05:32.363021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:32.362983 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 15:05:32.363235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:32.363052 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 15:05:33.146617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.146588 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk"] Oct 31 15:05:33.151813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.151794 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.223664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.223150 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk"] Oct 31 15:05:33.252946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.252920 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:05:33.253118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.253100 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:05:33.253195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.253180 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:05:33.260879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.260863 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:05:33.335136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.335102 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67xd\" (UniqueName: \"kubernetes.io/projected/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-kube-api-access-g67xd\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.335316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.335151 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.335365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.335323 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.363061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.363032 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 15:05:33.363191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.363079 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 15:05:33.436138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.436052 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.436319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.436155 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g67xd\" (UniqueName: \"kubernetes.io/projected/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-kube-api-access-g67xd\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.436319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.436183 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.436834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.436815 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.438662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.438644 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.521505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.521431 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67xd\" (UniqueName: \"kubernetes.io/projected/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-kube-api-access-g67xd\") pod \"maas-api-revoke-sub-8d69aa4d-hvqfk\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:33.766160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:33.766119 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:34.140935 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:05:34.140902 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2eaec2_31c9_47e0_93bd_1595ad194a3d.slice/crio-33de7ba12d2e7ddb0590e63a5cf5f411db319d33044f51b563946a867faf3fca WatchSource:0}: Error finding container 33de7ba12d2e7ddb0590e63a5cf5f411db319d33044f51b563946a867faf3fca: Status 404 returned error can't find the container with id 33de7ba12d2e7ddb0590e63a5cf5f411db319d33044f51b563946a867faf3fca Oct 31 15:05:34.141597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:34.141570 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk"] Oct 31 15:05:34.363080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:34.363047 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 15:05:34.363495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:34.363104 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 15:05:34.806380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:34.806352 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:34.987154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:34.987117 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" event={"ID":"1a2eaec2-31c9-47e0-93bd-1595ad194a3d","Type":"ContainerStarted","Data":"7ff248991d68c2bc2f51e4428dcdd8c3ea6b15921a95e53ebfd8cb1e252b3ef3"} Oct 31 15:05:34.987353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:34.987208 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" event={"ID":"1a2eaec2-31c9-47e0-93bd-1595ad194a3d","Type":"ContainerStarted","Data":"33de7ba12d2e7ddb0590e63a5cf5f411db319d33044f51b563946a867faf3fca"} Oct 31 15:05:35.109258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:35.109208 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" podStartSLOduration=2.109195388 podStartE2EDuration="2.109195388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:35.104563322 +0000 UTC m=+1180.724114735" watchObservedRunningTime="2026-10-31 15:05:35.109195388 +0000 UTC m=+1180.728746798" Oct 31 15:05:35.537393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:35.537285 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:35.539158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:35.539130 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:35.539652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:35.539630 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:05:35.794382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:35.794303 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:05:35.996692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:35.996655 4865 generic.go:356] "Generic (PLEG): container finished" podID="1a2eaec2-31c9-47e0-93bd-1595ad194a3d" containerID="7ff248991d68c2bc2f51e4428dcdd8c3ea6b15921a95e53ebfd8cb1e252b3ef3" exitCode=0 Oct 31 15:05:35.996692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:35.996687 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" event={"ID":"1a2eaec2-31c9-47e0-93bd-1595ad194a3d","Type":"ContainerDied","Data":"7ff248991d68c2bc2f51e4428dcdd8c3ea6b15921a95e53ebfd8cb1e252b3ef3"} Oct 31 15:05:37.070127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.065130 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb"] Oct 31 15:05:37.133497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.123662 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece07229-kfsvb"] Oct 31 15:05:37.154494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.149777 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj"] Oct 31 15:05:37.215540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.214222 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.250539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.236261 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj"] Oct 31 15:05:37.352555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.350598 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.352555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.350752 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.352555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.350807 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vqgj\" (UniqueName: \"kubernetes.io/projected/733e18fa-df5f-4218-8540-60bd58be1b02-kube-api-access-4vqgj\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.468497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.460933 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.468497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.461092 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.468497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.461149 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqgj\" (UniqueName: \"kubernetes.io/projected/733e18fa-df5f-4218-8540-60bd58be1b02-kube-api-access-4vqgj\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.521496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.520544 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.621540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.620873 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.653497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.651876 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqgj\" (UniqueName: \"kubernetes.io/projected/733e18fa-df5f-4218-8540-60bd58be1b02-kube-api-access-4vqgj\") pod \"maas-api-revoke-sub-a0db5cee-dzmqj\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.711134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.710859 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:37.830181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.828998 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:37.853911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.853597 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:37.868127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.867625 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:05:37.868127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.867944 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:37.953778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.952562 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:05:37.966505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:37.965576 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:05:38.341558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:38.329400 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:05:38.955570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:38.947680 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:38.957845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:38.957697 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:38.958214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:38.958150 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:05:38.995521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:38.971134 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:39.047525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.030135 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:39.227497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.227364 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:05:39.245678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.227662 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:39.245678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.227906 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:39.394639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.320500 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bde3625-e9ab-4d8d-84ae-b25d7a481bf3" path="/var/lib/kubelet/pods/9bde3625-e9ab-4d8d-84ae-b25d7a481bf3/volumes" Oct 31 15:05:39.571815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.571787 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:05:39.580604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.464943 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:05:39.634527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.606774 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:05:39.634527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.606829 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj"] Oct 31 15:05:39.634527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.608904 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:05:39.634527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.608960 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:05:39.634527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.608997 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:05:39.634527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.609032 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:05:39.634527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.609065 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:05:39.703003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:39.699611 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:40.110381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.110326 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:40.111941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.111913 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:05:40.158736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.158695 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:05:40.221493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.194987 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:05:40.221493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.195664 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:40.221493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.196342 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-kube-api-access-g67xd\" (UniqueName: \"kubernetes.io/projected/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-kube-api-access-g67xd\") pod \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " Oct 31 15:05:40.221493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.196427 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-openshift-ca\") pod \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " Oct 31 15:05:40.221493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.196586 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-xks-ca\") pod \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\" (UID: \"1a2eaec2-31c9-47e0-93bd-1595ad194a3d\") " Oct 31 15:05:40.225458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.225410 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-openshift-ca" pod "1a2eaec2-31c9-47e0-93bd-1595ad194a3d" (UID: "1a2eaec2-31c9-47e0-93bd-1595ad194a3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:40.237661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.237622 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:05:40.241619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.241566 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:05:40.266703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.266660 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:40.327748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.327710 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:40.412553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.400645 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-xks-ca" pod "1a2eaec2-31c9-47e0-93bd-1595ad194a3d" (UID: "1a2eaec2-31c9-47e0-93bd-1595ad194a3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:05:40.437497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.436740 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:40.548727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.457605 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:40.614494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.587699 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 31 15:05:40.661725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.603455 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-kube-api-access-g67xd" pod "1a2eaec2-31c9-47e0-93bd-1595ad194a3d" (UID: "1a2eaec2-31c9-47e0-93bd-1595ad194a3d"). InnerVolumeSpecName "kube-api-access-g67xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:40.685611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.632243 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:40.688692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.688661 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g67xd\" (UniqueName: \"kubernetes.io/projected/1a2eaec2-31c9-47e0-93bd-1595ad194a3d-kube-api-access-g67xd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:40.946569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:40.925141 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" event={"ID":"733e18fa-df5f-4218-8540-60bd58be1b02","Type":"ContainerStarted","Data":"9a339f4825555261e5daf50c565be9f84ae4b0c942d3f69f3a6e0d0ae2fbe589"} Oct 31 15:05:41.130513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.112728 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:41.173113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.171831 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:05:41.207707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.206103 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:41.207707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.206490 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:05:41.233948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.233915 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:41.241602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.241562 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:41.242198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.242169 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:41.314626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.314576 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:41.414013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.413978 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" event={"ID":"1a2eaec2-31c9-47e0-93bd-1595ad194a3d","Type":"ContainerDied","Data":"33de7ba12d2e7ddb0590e63a5cf5f411db319d33044f51b563946a867faf3fca"} Oct 31 15:05:41.414649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.414609 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33de7ba12d2e7ddb0590e63a5cf5f411db319d33044f51b563946a867faf3fca" Oct 31 15:05:41.414957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.414944 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk" Oct 31 15:05:41.689601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:41.689557 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.115560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.114602 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.174849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.174789 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.187676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.187578 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.214079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.212583 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:05:42.214079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.212982 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:05:42.245613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.245564 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.246185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.246155 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:05:42.316626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.316570 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:05:42.461550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.450497 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" event={"ID":"733e18fa-df5f-4218-8540-60bd58be1b02","Type":"ContainerStarted","Data":"6aac90021c84e75b78f0d047be6a2e4aac6f0b2572bb706022627e4fb721f035"} Oct 31 15:05:42.572762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.572728 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:42.573026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.572997 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.618403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.618337 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" podStartSLOduration=5.618317403 podStartE2EDuration="5.618317403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:42.618273708 +0000 UTC m=+1188.237825119" watchObservedRunningTime="2026-10-31 15:05:42.618317403 +0000 UTC m=+1188.237868816" Oct 31 15:05:42.880382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.707832 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.980114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:42.980081 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:05:43.153814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.153726 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:05:43.189157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.189116 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.228789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.228719 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.246711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.246672 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:05:43.255862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.246635 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.274635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.274594 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.333767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.310678 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:05:43.346096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.346067 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:05:43.381490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.379374 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.793700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:43.793672 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:05:44.087416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.087328 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp"] Oct 31 15:05:44.088490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.088443 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1a2eaec2-31c9-47e0-93bd-1595ad194a3d" containerName="revoke-keys" Oct 31 15:05:44.088655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.088638 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2eaec2-31c9-47e0-93bd-1595ad194a3d" containerName="revoke-keys" Oct 31 15:05:44.089148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.089105 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2eaec2-31c9-47e0-93bd-1595ad194a3d" containerName="revoke-keys" Oct 31 15:05:44.144496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.143569 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.241495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.230911 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.241495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.231002 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8pl\" (UniqueName: \"kubernetes.io/projected/98489ae7-ed99-4396-93a6-bd58892f8e17-kube-api-access-wm8pl\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.241495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.231186 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.285489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.267869 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:05:44.342550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.336832 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.343012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.342979 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.343173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.343155 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8pl\" (UniqueName: \"kubernetes.io/projected/98489ae7-ed99-4396-93a6-bd58892f8e17-kube-api-access-wm8pl\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.343752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.343722 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:05:44.358491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.345907 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.408698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.378914 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:05:44.504602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.504567 4865 generic.go:356] "Generic (PLEG): container finished" podID="733e18fa-df5f-4218-8540-60bd58be1b02" containerID="6aac90021c84e75b78f0d047be6a2e4aac6f0b2572bb706022627e4fb721f035" exitCode=0 Oct 31 15:05:44.584903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.584869 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.663538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.652410 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8pl\" (UniqueName: \"kubernetes.io/projected/98489ae7-ed99-4396-93a6-bd58892f8e17-kube-api-access-wm8pl\") pod \"maas-api-revoke-sub-65a747ae-hmkzp\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:44.766495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.763637 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp"] Oct 31 15:05:44.766495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.763733 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:05:44.766495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.763754 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" event={"ID":"733e18fa-df5f-4218-8540-60bd58be1b02","Type":"ContainerDied","Data":"6aac90021c84e75b78f0d047be6a2e4aac6f0b2572bb706022627e4fb721f035"} Oct 31 15:05:44.766495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.763802 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:05:44.852564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:44.852528 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:45.247736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.247692 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp"] Oct 31 15:05:45.514691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.514655 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" event={"ID":"98489ae7-ed99-4396-93a6-bd58892f8e17","Type":"ContainerStarted","Data":"020ff2fb199c0043d95ac7c989f329ce814b0c76b3c6f5f702d900ad3c6b526e"} Oct 31 15:05:45.631260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.631206 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" podStartSLOduration=2.631189912 podStartE2EDuration="2.631189912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:45.630095723 +0000 UTC m=+1191.249647137" watchObservedRunningTime="2026-10-31 15:05:45.631189912 +0000 UTC m=+1191.250741322" Oct 31 15:05:45.670287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.670260 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:45.772531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.772421 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-openshift-ca\") pod \"733e18fa-df5f-4218-8540-60bd58be1b02\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " Oct 31 15:05:45.772682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.772570 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733e18fa-df5f-4218-8540-60bd58be1b02-kube-api-access-4vqgj\" (UniqueName: \"kubernetes.io/projected/733e18fa-df5f-4218-8540-60bd58be1b02-kube-api-access-4vqgj\") pod \"733e18fa-df5f-4218-8540-60bd58be1b02\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " Oct 31 15:05:45.772720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.772713 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-xks-ca\") pod \"733e18fa-df5f-4218-8540-60bd58be1b02\" (UID: \"733e18fa-df5f-4218-8540-60bd58be1b02\") " Oct 31 15:05:45.772766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.772744 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-openshift-ca" pod "733e18fa-df5f-4218-8540-60bd58be1b02" (UID: "733e18fa-df5f-4218-8540-60bd58be1b02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:45.773154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.773136 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:45.774841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.774812 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-xks-ca" pod "733e18fa-df5f-4218-8540-60bd58be1b02" (UID: "733e18fa-df5f-4218-8540-60bd58be1b02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:05:45.774938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.774850 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733e18fa-df5f-4218-8540-60bd58be1b02-kube-api-access-4vqgj" pod "733e18fa-df5f-4218-8540-60bd58be1b02" (UID: "733e18fa-df5f-4218-8540-60bd58be1b02"). InnerVolumeSpecName "kube-api-access-4vqgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:45.874536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.874499 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vqgj\" (UniqueName: \"kubernetes.io/projected/733e18fa-df5f-4218-8540-60bd58be1b02-kube-api-access-4vqgj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:45.874536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:45.874530 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733e18fa-df5f-4218-8540-60bd58be1b02-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:46.523953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:46.523920 4865 generic.go:356] "Generic (PLEG): container finished" podID="98489ae7-ed99-4396-93a6-bd58892f8e17" containerID="62af61d4ea03d2a7aea361552c48e92e30666434109f25083e9deaa2313a5efd" exitCode=0 Oct 31 15:05:46.524103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:46.524002 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" event={"ID":"98489ae7-ed99-4396-93a6-bd58892f8e17","Type":"ContainerDied","Data":"62af61d4ea03d2a7aea361552c48e92e30666434109f25083e9deaa2313a5efd"} Oct 31 15:05:46.525394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:46.525376 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" Oct 31 15:05:46.525551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:46.525407 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj" event={"ID":"733e18fa-df5f-4218-8540-60bd58be1b02","Type":"ContainerDied","Data":"9a339f4825555261e5daf50c565be9f84ae4b0c942d3f69f3a6e0d0ae2fbe589"} Oct 31 15:05:46.525551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:46.525443 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a339f4825555261e5daf50c565be9f84ae4b0c942d3f69f3a6e0d0ae2fbe589" Oct 31 15:05:46.628261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:46.628234 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:47.490109 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:05:47.490070 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:05:47.691324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.691297 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:47.799239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.799202 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-openshift-ca\") pod \"98489ae7-ed99-4396-93a6-bd58892f8e17\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " Oct 31 15:05:47.799239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.799242 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98489ae7-ed99-4396-93a6-bd58892f8e17-kube-api-access-wm8pl\" (UniqueName: \"kubernetes.io/projected/98489ae7-ed99-4396-93a6-bd58892f8e17-kube-api-access-wm8pl\") pod \"98489ae7-ed99-4396-93a6-bd58892f8e17\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " Oct 31 15:05:47.799461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.799353 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-xks-ca\") pod \"98489ae7-ed99-4396-93a6-bd58892f8e17\" (UID: \"98489ae7-ed99-4396-93a6-bd58892f8e17\") " Oct 31 15:05:47.799670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.799646 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-openshift-ca" pod "98489ae7-ed99-4396-93a6-bd58892f8e17" (UID: "98489ae7-ed99-4396-93a6-bd58892f8e17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:47.799941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.799924 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:47.801517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.801493 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-xks-ca" pod "98489ae7-ed99-4396-93a6-bd58892f8e17" (UID: "98489ae7-ed99-4396-93a6-bd58892f8e17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:05:47.801593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.801572 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98489ae7-ed99-4396-93a6-bd58892f8e17-kube-api-access-wm8pl" pod "98489ae7-ed99-4396-93a6-bd58892f8e17" (UID: "98489ae7-ed99-4396-93a6-bd58892f8e17"). InnerVolumeSpecName "kube-api-access-wm8pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:05:47.901675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.901597 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8pl\" (UniqueName: \"kubernetes.io/projected/98489ae7-ed99-4396-93a6-bd58892f8e17-kube-api-access-wm8pl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:47.901675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:47.901627 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98489ae7-ed99-4396-93a6-bd58892f8e17-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:05:48.510687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.510658 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64"] Oct 31 15:05:48.511196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.511183 4865 cpu_manager.go:414] "containerMap: removing container" podUID="98489ae7-ed99-4396-93a6-bd58892f8e17" containerName="revoke-keys" Oct 31 15:05:48.511196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.511196 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98489ae7-ed99-4396-93a6-bd58892f8e17" containerName="revoke-keys" Oct 31 15:05:48.511300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.511215 4865 cpu_manager.go:414] "containerMap: removing container" podUID="733e18fa-df5f-4218-8540-60bd58be1b02" containerName="revoke-keys" Oct 31 15:05:48.511300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.511223 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733e18fa-df5f-4218-8540-60bd58be1b02" containerName="revoke-keys" Oct 31 15:05:48.511403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.511388 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="98489ae7-ed99-4396-93a6-bd58892f8e17" containerName="revoke-keys" Oct 31 15:05:48.511444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.511408 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="733e18fa-df5f-4218-8540-60bd58be1b02" containerName="revoke-keys" Oct 31 15:05:48.514852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.514836 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.539990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.539963 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" Oct 31 15:05:48.539990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.539962 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp" event={"ID":"98489ae7-ed99-4396-93a6-bd58892f8e17","Type":"ContainerDied","Data":"020ff2fb199c0043d95ac7c989f329ce814b0c76b3c6f5f702d900ad3c6b526e"} Oct 31 15:05:48.540147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.540017 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="020ff2fb199c0043d95ac7c989f329ce814b0c76b3c6f5f702d900ad3c6b526e" Oct 31 15:05:48.574617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.574581 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64"] Oct 31 15:05:48.611951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.611922 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zm4n\" (UniqueName: \"kubernetes.io/projected/62479a79-197f-4a3f-9c53-bb3e9741fb86-kube-api-access-4zm4n\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.612126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.611977 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62479a79-197f-4a3f-9c53-bb3e9741fb86-tls-certs\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.612188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.612143 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-model-cache\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.612269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.612236 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-tmp-dir\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.612341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.612326 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-dshm\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.612499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.612465 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-home\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.618111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.618090 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"distinct-2-w1-fb2348-kserve-self-signed-certs\"" Oct 31 15:05:48.680504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.680435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:48.716281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.715377 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-home\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.716281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.715525 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zm4n\" (UniqueName: \"kubernetes.io/projected/62479a79-197f-4a3f-9c53-bb3e9741fb86-kube-api-access-4zm4n\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.716281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.715584 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62479a79-197f-4a3f-9c53-bb3e9741fb86-tls-certs\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.716281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.715666 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-model-cache\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.716281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.715748 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-tmp-dir\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.716281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.715790 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-dshm\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.717542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.717515 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-home\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.725354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.725322 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-model-cache\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.725937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.725910 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-tmp-dir\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.756405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.756300 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62479a79-197f-4a3f-9c53-bb3e9741fb86-tls-certs\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.767703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.767669 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-dshm\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.786382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.786349 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zm4n\" (UniqueName: \"kubernetes.io/projected/62479a79-197f-4a3f-9c53-bb3e9741fb86-kube-api-access-4zm4n\") pod \"distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:48.827110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:48.827082 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:49.260961 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:05:49.260934 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62479a79_197f_4a3f_9c53_bb3e9741fb86.slice/crio-d85e881bc9f8e28235a2145093203af543c2320253f417d85e734739df5899b7 WatchSource:0}: Error finding container d85e881bc9f8e28235a2145093203af543c2320253f417d85e734739df5899b7: Status 404 returned error can't find the container with id d85e881bc9f8e28235a2145093203af543c2320253f417d85e734739df5899b7 Oct 31 15:05:49.262617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:49.262591 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64"] Oct 31 15:05:49.549494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:49.549384 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" event={"ID":"62479a79-197f-4a3f-9c53-bb3e9741fb86","Type":"ContainerStarted","Data":"052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e"} Oct 31 15:05:49.549494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:49.549426 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" event={"ID":"62479a79-197f-4a3f-9c53-bb3e9741fb86","Type":"ContainerStarted","Data":"d85e881bc9f8e28235a2145093203af543c2320253f417d85e734739df5899b7"} Oct 31 15:05:49.665476 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:49.665420 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podStartSLOduration=1.665406184 podStartE2EDuration="1.665406184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:49.664824085 +0000 UTC m=+1195.284375496" watchObservedRunningTime="2026-10-31 15:05:49.665406184 +0000 UTC m=+1195.284957594" Oct 31 15:05:49.836545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:49.832581 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:51.316673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:51.316643 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:51.488827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:51.484498 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:52.508721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:52.508692 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:53.082749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:53.082719 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2"] Oct 31 15:05:53.082963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:53.082943 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" containerID="cri-o://14005d1aa158119ecd38926258dc93fbacdf1e97ac807dc8b42f73e076ada2a1" gracePeriod=30 Oct 31 15:05:53.151760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:53.151728 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz"] Oct 31 15:05:53.152006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:53.151977 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" containerID="cri-o://3be40b659000f55515a8c2441ef66f16762502403a45d18310e860d49b0def1e" gracePeriod=30 Oct 31 15:05:54.643846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:54.643817 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:05:54.653282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:54.653261 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:05:54.653541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:54.653526 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:05:54.661107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:54.661091 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:05:54.803932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:54.803905 4865 scope.go:122] "RemoveContainer" containerID="d6065373051993c58a62bc1d4acccc09598b4b12e80ffb312ceab374c62ce4aa" Oct 31 15:05:54.814008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:54.813984 4865 scope.go:122] "RemoveContainer" containerID="36ff0bee4fb19e909c97d969ae78ebde7e9f5e338b04a3cef198cb317bcc5e50" Oct 31 15:05:56.123711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:56.123666 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:05:56.273910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:56.273876 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:05:56.274154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:56.274127 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:05:58.698174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:58.698080 4865 generic.go:356] "Generic (PLEG): container finished" podID="88408248-6fd6-4741-9979-f25c6648056b" containerID="14005d1aa158119ecd38926258dc93fbacdf1e97ac807dc8b42f73e076ada2a1" exitCode=0 Oct 31 15:05:58.699087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:58.699058 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" event={"ID":"88408248-6fd6-4741-9979-f25c6648056b","Type":"ContainerDied","Data":"14005d1aa158119ecd38926258dc93fbacdf1e97ac807dc8b42f73e076ada2a1"} Oct 31 15:05:58.746820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:58.746782 4865 generic.go:356] "Generic (PLEG): container finished" podID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerID="3be40b659000f55515a8c2441ef66f16762502403a45d18310e860d49b0def1e" exitCode=0 Oct 31 15:05:58.747009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:58.746884 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" event={"ID":"7abf9c3e-ff1e-4127-9f44-da9afef94b65","Type":"ContainerDied","Data":"3be40b659000f55515a8c2441ef66f16762502403a45d18310e860d49b0def1e"} Oct 31 15:05:58.747009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:58.746942 4865 scope.go:122] "RemoveContainer" containerID="69b2bfdc0012a409cecdff92eae76fa40306c4a1d373f88f6df130f409097875" Oct 31 15:05:58.832805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:58.832769 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:05:59.744496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.743520 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:05:59.874552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.874197 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" event={"ID":"88408248-6fd6-4741-9979-f25c6648056b","Type":"ContainerDied","Data":"aac1509f71d8ba4f3c410fc9ef4e716eb16cfe98811908bdf5d06652cd9bd596"} Oct 31 15:05:59.874552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.874285 4865 scope.go:122] "RemoveContainer" containerID="14005d1aa158119ecd38926258dc93fbacdf1e97ac807dc8b42f73e076ada2a1" Oct 31 15:05:59.874552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.874393 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" Oct 31 15:05:59.912500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.906801 4865 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930574 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-envoy\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930664 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-kube-api-access-2sjjs\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930706 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-credential-socket\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930761 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-socket\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930799 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88408248-6fd6-4741-9979-f25c6648056b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88408248-6fd6-4741-9979-f25c6648056b-istiod-ca-cert\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930843 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-certs\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930929 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88408248-6fd6-4741-9979-f25c6648056b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88408248-6fd6-4741-9979-f25c6648056b-istio-podinfo\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.930966 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-istio-token\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-istio-token\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.943604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.931055 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-data\") pod \"88408248-6fd6-4741-9979-f25c6648056b\" (UID: \"88408248-6fd6-4741-9979-f25c6648056b\") " Oct 31 15:05:59.977528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.959012 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-socket" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:05:59.977528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.959332 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-credential-socket" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:05:59.977528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.960077 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-certs" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:05:59.977528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.960599 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88408248-6fd6-4741-9979-f25c6648056b-istiod-ca-cert" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:05:59.977528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.960662 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-data" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:06:00.005536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:05:59.997718 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88408248-6fd6-4741-9979-f25c6648056b-istio-podinfo" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.033778 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88408248-6fd6-4741-9979-f25c6648056b-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.033813 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.033831 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.033846 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.033860 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88408248-6fd6-4741-9979-f25c6648056b-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.033875 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.040599 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:06:00.068525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.040680 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.121680 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.128677 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-envoy" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.129409 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-istio-token" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.129524 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-kube-api-access-2sjjs" pod "88408248-6fd6-4741-9979-f25c6648056b" (UID: "88408248-6fd6-4741-9979-f25c6648056b"). InnerVolumeSpecName "kube-api-access-2sjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.137498 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.137531 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88408248-6fd6-4741-9979-f25c6648056b-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.137548 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/88408248-6fd6-4741-9979-f25c6648056b-kube-api-access-2sjjs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.218372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.197775 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.261214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.257421 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.262863 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-credential-socket\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.262936 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-kube-api-access-227v8\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-kube-api-access-227v8\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.262986 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-envoy\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.263024 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-podinfo\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.263087 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-socket\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.263133 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-certs\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.263172 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-token\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-token\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.263350 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-data\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.275791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.263410 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istiod-ca-cert\") pod \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\" (UID: \"7abf9c3e-ff1e-4127-9f44-da9afef94b65\") " Oct 31 15:06:00.303782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.290750 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istiod-ca-cert" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:06:00.303782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.291119 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-credential-socket" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:06:00.303782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.295573 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-socket" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:06:00.307503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.306708 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-certs" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:06:00.316607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.312247 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-token" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:00.348494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.330217 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-data" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:06:00.348494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.330310 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.348494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.340191 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.376138 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.376945 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.378283 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.378308 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.378322 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.378334 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.378346 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.386495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.378358 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.416222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.400951 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:06:00.416222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.416130 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.428716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.418014 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-podinfo" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:06:00.447387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.436594 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-envoy" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:06:00.447387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.436704 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-kube-api-access-227v8" pod "7abf9c3e-ff1e-4127-9f44-da9afef94b65" (UID: "7abf9c3e-ff1e-4127-9f44-da9afef94b65"). InnerVolumeSpecName "kube-api-access-227v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:00.767554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.766581 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 31 15:06:00.775495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.772731 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.834697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.802622 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-227v8\" (UniqueName: \"kubernetes.io/projected/7abf9c3e-ff1e-4127-9f44-da9afef94b65-kube-api-access-227v8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.834697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.802655 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:00.834697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:00.802671 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7abf9c3e-ff1e-4127-9f44-da9afef94b65-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:01.106546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.103511 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2"] Oct 31 15:06:01.106546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.103562 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-k42w2"] Oct 31 15:06:01.123496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.120746 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" Oct 31 15:06:01.123496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.121016 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz" event={"ID":"7abf9c3e-ff1e-4127-9f44-da9afef94b65","Type":"ContainerDied","Data":"0692bc6551d245224e451cf54bfb2e50a2935cd3dbf24be1fdc171deb05c1c34"} Oct 31 15:06:01.123496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.121075 4865 scope.go:122] "RemoveContainer" containerID="3be40b659000f55515a8c2441ef66f16762502403a45d18310e860d49b0def1e" Oct 31 15:06:01.200855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.200802 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.201780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.201754 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:06:01.266612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.266562 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:06:01.276497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.267255 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:06:01.350536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.334546 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.350536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.347621 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:06:01.350536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.350043 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.350536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.350415 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:06:01.378661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.378561 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.404497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.379200 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:06:01.412495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.382580 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:06:01.412802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.412782 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:06:01.433305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.408298 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.449113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.449086 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:06:01.788496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.775765 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.788496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:01.776103 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:06:02.022505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.021586 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:06:02.022505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.021675 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.196858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.193459 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.201694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.201654 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.285497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.276789 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.326806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.309136 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:02.335729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.335686 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.390629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.390581 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.411639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.411591 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.439496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.413824 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:06:02.481456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.465782 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:06:02.902179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.902097 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88408248-6fd6-4741-9979-f25c6648056b" path="/var/lib/kubelet/pods/88408248-6fd6-4741-9979-f25c6648056b/volumes" Oct 31 15:06:02.903295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.903278 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:06:02.903507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.903493 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:06:02.905247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.905232 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:06:02.905417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.905404 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:06:02.905757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.905743 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:06:02.906138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.906124 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:06:02.906363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.906349 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:06:02.906519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:02.906506 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:06:03.496030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.495997 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs"] Oct 31 15:06:03.496817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.496794 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" Oct 31 15:06:03.496907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.496819 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" Oct 31 15:06:03.496907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.496860 4865 cpu_manager.go:414] "containerMap: removing container" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" Oct 31 15:06:03.496907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.496868 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" Oct 31 15:06:03.497061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.497047 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" Oct 31 15:06:03.497099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.497068 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" Oct 31 15:06:03.497099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.497082 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="88408248-6fd6-4741-9979-f25c6648056b" containerName="istio-proxy" Oct 31 15:06:03.510399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.509921 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.525168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.525132 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.525353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.525324 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.525428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.525390 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/5d44e54a-4c49-4cb3-9b76-239d210ee158-kube-api-access-mctxp\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.557017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.556992 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs"] Oct 31 15:06:03.564044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.564009 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:06:03.564188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.564145 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:06:03.564188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.564145 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:06:03.564306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.564198 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:06:03.626704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.626669 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.626895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.626819 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.626895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.626872 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/5d44e54a-4c49-4cb3-9b76-239d210ee158-kube-api-access-mctxp\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.627393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.627372 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.629447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.629423 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.684832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.684802 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/5d44e54a-4c49-4cb3-9b76-239d210ee158-kube-api-access-mctxp\") pod \"maas-api-revoke-sub-769ddd41-t9kgs\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.828611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.828527 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:03.991657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.990421 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz"] Oct 31 15:06:03.994810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:03.994780 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-98lbz"] Oct 31 15:06:04.336954 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:04.336926 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d44e54a_4c49_4cb3_9b76_239d210ee158.slice/crio-f677796bd96c2fe88851fe3405f7291018cdc671ab7c9854acaed31cb2459f5a WatchSource:0}: Error finding container f677796bd96c2fe88851fe3405f7291018cdc671ab7c9854acaed31cb2459f5a: Status 404 returned error can't find the container with id f677796bd96c2fe88851fe3405f7291018cdc671ab7c9854acaed31cb2459f5a Oct 31 15:06:04.338084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:04.338060 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs"] Oct 31 15:06:04.498377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:04.498350 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" path="/var/lib/kubelet/pods/7abf9c3e-ff1e-4127-9f44-da9afef94b65/volumes" Oct 31 15:06:04.526827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:04.526802 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" event={"ID":"5d44e54a-4c49-4cb3-9b76-239d210ee158","Type":"ContainerStarted","Data":"f677796bd96c2fe88851fe3405f7291018cdc671ab7c9854acaed31cb2459f5a"} Oct 31 15:06:05.535515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:05.535404 4865 generic.go:356] "Generic (PLEG): container finished" podID="5d44e54a-4c49-4cb3-9b76-239d210ee158" containerID="4590a92f034df34809ecfc5be1bbd27300380402dc61864287159f9459843a44" exitCode=0 Oct 31 15:06:05.535887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:05.535513 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" event={"ID":"5d44e54a-4c49-4cb3-9b76-239d210ee158","Type":"ContainerDied","Data":"4590a92f034df34809ecfc5be1bbd27300380402dc61864287159f9459843a44"} Oct 31 15:06:06.732906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.732882 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:06.782157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.782128 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-openshift-ca\") pod \"5d44e54a-4c49-4cb3-9b76-239d210ee158\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " Oct 31 15:06:06.782397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.782191 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-xks-ca\") pod \"5d44e54a-4c49-4cb3-9b76-239d210ee158\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " Oct 31 15:06:06.782397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.782290 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d44e54a-4c49-4cb3-9b76-239d210ee158-kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/5d44e54a-4c49-4cb3-9b76-239d210ee158-kube-api-access-mctxp\") pod \"5d44e54a-4c49-4cb3-9b76-239d210ee158\" (UID: \"5d44e54a-4c49-4cb3-9b76-239d210ee158\") " Oct 31 15:06:06.782605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.782581 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-openshift-ca" pod "5d44e54a-4c49-4cb3-9b76-239d210ee158" (UID: "5d44e54a-4c49-4cb3-9b76-239d210ee158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:06:06.782828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.782813 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:06.784519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.784496 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-xks-ca" pod "5d44e54a-4c49-4cb3-9b76-239d210ee158" (UID: "5d44e54a-4c49-4cb3-9b76-239d210ee158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:06:06.784658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.784637 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d44e54a-4c49-4cb3-9b76-239d210ee158-kube-api-access-mctxp" pod "5d44e54a-4c49-4cb3-9b76-239d210ee158" (UID: "5d44e54a-4c49-4cb3-9b76-239d210ee158"). InnerVolumeSpecName "kube-api-access-mctxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:06.884673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.884594 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d44e54a-4c49-4cb3-9b76-239d210ee158-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:06.884673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.884632 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/5d44e54a-4c49-4cb3-9b76-239d210ee158-kube-api-access-mctxp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:06.913802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:06.913770 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:07.047805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:07.047764 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:07.048603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:07.048585 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx"] Oct 31 15:06:07.051882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:07.051861 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35fcf76c-hq5fx"] Oct 31 15:06:07.551220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:07.551194 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" Oct 31 15:06:07.551419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:07.551223 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs" event={"ID":"5d44e54a-4c49-4cb3-9b76-239d210ee158","Type":"ContainerDied","Data":"f677796bd96c2fe88851fe3405f7291018cdc671ab7c9854acaed31cb2459f5a"} Oct 31 15:06:07.551419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:07.551281 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f677796bd96c2fe88851fe3405f7291018cdc671ab7c9854acaed31cb2459f5a" Oct 31 15:06:08.498830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:08.498796 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d537238-fd1d-41b5-8549-e58c22b41fa9" path="/var/lib/kubelet/pods/0d537238-fd1d-41b5-8549-e58c22b41fa9/volumes" Oct 31 15:06:08.839375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:08.839294 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:06:08.851385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:08.851358 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:06:10.779079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:10.779040 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:06:11.787495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:11.786456 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:12.441999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.441936 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq"] Oct 31 15:06:12.443067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.443041 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5d44e54a-4c49-4cb3-9b76-239d210ee158" containerName="revoke-keys" Oct 31 15:06:12.443245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.443229 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d44e54a-4c49-4cb3-9b76-239d210ee158" containerName="revoke-keys" Oct 31 15:06:12.443404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.443391 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" Oct 31 15:06:12.443510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.443498 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abf9c3e-ff1e-4127-9f44-da9afef94b65" containerName="istio-proxy" Oct 31 15:06:12.443746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.443733 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d44e54a-4c49-4cb3-9b76-239d210ee158" containerName="revoke-keys" Oct 31 15:06:12.522766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.522734 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.731785 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wr6k\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-kube-api-access-2wr6k\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.731861 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.731982 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35246052-d285-42ea-9fbf-5ff3f818d064-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.732030 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.732086 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.732128 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35246052-d285-42ea-9fbf-5ff3f818d064-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.748690 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.748859 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.751260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.748901 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.822737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.811554 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq"] Oct 31 15:06:12.848502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.847568 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:12.855600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.855569 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.912557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.912527 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.912946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.912924 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wr6k\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-kube-api-access-2wr6k\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.913193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.913169 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.913564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.913543 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35246052-d285-42ea-9fbf-5ff3f818d064-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.913776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.913739 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.913965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.913945 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.914143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.914126 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35246052-d285-42ea-9fbf-5ff3f818d064-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.914364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.914345 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.915152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.915128 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.915293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.911994 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:12.915996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.915974 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.927351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.927325 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.928450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.928426 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35246052-d285-42ea-9fbf-5ff3f818d064-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:12.928889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:12.928868 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:13.059005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.058824 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:13.062789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.059349 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35246052-d285-42ea-9fbf-5ff3f818d064-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:13.081631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.081590 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:13.099625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.099592 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wr6k\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-kube-api-access-2wr6k\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:13.123500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.121022 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4ctjq\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:13.205863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.205829 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:13.413726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.413632 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:13.640986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.640545 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:13.829507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.829320 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:13.830013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.829781 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:06:13.859505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.858223 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:13.859505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.858608 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:06:13.883586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.866121 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:06:13.925612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.919824 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:13.925612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:13.920238 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:06:14.170492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.167075 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:06:14.170492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.167720 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:06:14.350486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.349815 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:06:14.351264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.351237 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:14.432505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.431286 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:06:14.434231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.433367 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:06:14.548834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.548790 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:14.773518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.773456 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:14.832521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.792628 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:14.862722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.862635 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:14.951145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.951113 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:06:14.974196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:14.971539 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:06:15.027567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.016419 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:15.027567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.016539 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:15.027567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.016587 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:15.028483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.028445 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:06:15.028677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.028660 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq"] Oct 31 15:06:15.144052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.144024 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:06:15.144439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.144419 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:06:15.146861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.146834 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" event={"ID":"35246052-d285-42ea-9fbf-5ff3f818d064","Type":"ContainerStarted","Data":"0cfd5c3c12132da809216abf5df1462615cf4d4fb64d1d82689aac97177cc6ef"} Oct 31 15:06:15.294403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.294314 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:06:15.658547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:15.634597 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:06:16.185581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.185384 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:16.197493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.193526 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" event={"ID":"35246052-d285-42ea-9fbf-5ff3f818d064","Type":"ContainerStarted","Data":"5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25"} Oct 31 15:06:16.208958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.208934 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:16.214624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.214599 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:16.220821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.220790 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:16.362001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.361965 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:16.382176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.369427 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podStartSLOduration=4.369407007 podStartE2EDuration="4.369407007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:16.361970399 +0000 UTC m=+1221.981521815" watchObservedRunningTime="2026-10-31 15:06:16.369407007 +0000 UTC m=+1221.988958416" Oct 31 15:06:16.715720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.715682 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m"] Oct 31 15:06:16.720038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.720002 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.782063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.774866 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"distinct-w0-56bc96-kserve-self-signed-certs\"" Oct 31 15:06:16.784145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.783947 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m"] Oct 31 15:06:16.786616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.786506 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmfv8\" (UniqueName: \"kubernetes.io/projected/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-kube-api-access-hmfv8\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.786841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.786826 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-model-cache\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.787022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.787004 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tls-certs\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.787161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.787144 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-home\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.787365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.787348 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-dshm\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.787524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.787508 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tmp-dir\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.818650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.818600 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:16.899192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.899160 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tmp-dir\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.899548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.899525 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmfv8\" (UniqueName: \"kubernetes.io/projected/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-kube-api-access-hmfv8\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.899726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.899711 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-model-cache\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.899881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.899867 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tls-certs\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.900004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.899990 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-home\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.900229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.900209 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-dshm\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.901664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.900726 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-model-cache\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.901795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.900927 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tmp-dir\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.902298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.902278 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-home\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.989827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.981426 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-dshm\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:16.990654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:16.990624 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tls-certs\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:17.010491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:17.007963 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmfv8\" (UniqueName: \"kubernetes.io/projected/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-kube-api-access-hmfv8\") pod \"distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:17.047848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:17.047813 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:17.215577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:17.207076 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:17.349715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:17.346625 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:17.491056 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:06:17.491014 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:06:17.576760 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:17.576730 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod078f1c4f_9450_4770_ab7b_0ff3bfcd0f66.slice/crio-5d85c959d44f1d3af55ba68074aa2660a1c99a42a0a857ff098aa34fb3d50f4c WatchSource:0}: Error finding container 5d85c959d44f1d3af55ba68074aa2660a1c99a42a0a857ff098aa34fb3d50f4c: Status 404 returned error can't find the container with id 5d85c959d44f1d3af55ba68074aa2660a1c99a42a0a857ff098aa34fb3d50f4c Oct 31 15:06:17.577840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:17.577812 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m"] Oct 31 15:06:18.218806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:18.218773 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:18.219352 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:18.218824 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:18.364312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:18.364274 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" event={"ID":"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66","Type":"ContainerStarted","Data":"dc8cb268d8f785345c29df9afe6ad77ba43ccf6ce0ea62373175399bb269e656"} Oct 31 15:06:18.364534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:18.364324 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" event={"ID":"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66","Type":"ContainerStarted","Data":"5d85c959d44f1d3af55ba68074aa2660a1c99a42a0a857ff098aa34fb3d50f4c"} Oct 31 15:06:18.481827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:18.481717 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podStartSLOduration=2.481703627 podStartE2EDuration="2.481703627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:18.480892834 +0000 UTC m=+1224.100444245" watchObservedRunningTime="2026-10-31 15:06:18.481703627 +0000 UTC m=+1224.101255016" Oct 31 15:06:18.876886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:18.876803 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:19.049500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:19.049435 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:19.216565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:19.214952 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:19.216565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:19.215012 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:20.211494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:20.210854 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:20.211494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:20.210916 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:21.210504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:21.209991 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:21.210504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:21.210050 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:22.018072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.018021 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh"] Oct 31 15:06:22.023794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.023771 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.067644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.067613 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh"] Oct 31 15:06:22.077378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.077353 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:06:22.115918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.115733 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:06:22.115918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.115832 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:06:22.115918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.115860 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:06:22.149894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.149863 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.150092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.150060 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.150154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.150102 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krc9\" (UniqueName: \"kubernetes.io/projected/2aa6534b-4e97-4111-8c31-c1bfadee55ac-kube-api-access-7krc9\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.215507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.214886 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:22.215507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.214948 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:22.251198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.251145 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.251405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.251305 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.251405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.251355 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krc9\" (UniqueName: \"kubernetes.io/projected/2aa6534b-4e97-4111-8c31-c1bfadee55ac-kube-api-access-7krc9\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.252012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.251987 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.253770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.253752 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.321666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.321585 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krc9\" (UniqueName: \"kubernetes.io/projected/2aa6534b-4e97-4111-8c31-c1bfadee55ac-kube-api-access-7krc9\") pod \"maas-api-revoke-sub-6569e34c-fbdjh\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.337791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.337764 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:22.683907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.683822 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:22.729315 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:22.729283 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa6534b_4e97_4111_8c31_c1bfadee55ac.slice/crio-619030ed0a1cddd54ff34c8896d76f84394ba99a7a37a63a716590bb22774f8a WatchSource:0}: Error finding container 619030ed0a1cddd54ff34c8896d76f84394ba99a7a37a63a716590bb22774f8a: Status 404 returned error can't find the container with id 619030ed0a1cddd54ff34c8896d76f84394ba99a7a37a63a716590bb22774f8a Oct 31 15:06:22.729437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.729354 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh"] Oct 31 15:06:22.818058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:22.818026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:23.206620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:23.206578 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 15:06:23.206993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:23.206644 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 15:06:23.412353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:23.412272 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" event={"ID":"2aa6534b-4e97-4111-8c31-c1bfadee55ac","Type":"ContainerStarted","Data":"907a3436be80713c4d1a96c75e0ca5628c4f1f6c1030659db9119730b3d57c3a"} Oct 31 15:06:23.412353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:23.412325 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" event={"ID":"2aa6534b-4e97-4111-8c31-c1bfadee55ac","Type":"ContainerStarted","Data":"619030ed0a1cddd54ff34c8896d76f84394ba99a7a37a63a716590bb22774f8a"} Oct 31 15:06:23.533933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:23.533880 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" podStartSLOduration=2.533864797 podStartE2EDuration="2.533864797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:23.531114762 +0000 UTC m=+1229.150666197" watchObservedRunningTime="2026-10-31 15:06:23.533864797 +0000 UTC m=+1229.153416207" Oct 31 15:06:23.711339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:23.711309 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:24.281739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.281709 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:24.282193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.282070 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:24.283027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.283002 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:06:24.421303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.421272 4865 generic.go:356] "Generic (PLEG): container finished" podID="2aa6534b-4e97-4111-8c31-c1bfadee55ac" containerID="907a3436be80713c4d1a96c75e0ca5628c4f1f6c1030659db9119730b3d57c3a" exitCode=0 Oct 31 15:06:24.421516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.421360 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" event={"ID":"2aa6534b-4e97-4111-8c31-c1bfadee55ac","Type":"ContainerDied","Data":"907a3436be80713c4d1a96c75e0ca5628c4f1f6c1030659db9119730b3d57c3a"} Oct 31 15:06:24.553970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.553896 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv"] Oct 31 15:06:24.557955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.557934 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.587900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.587877 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f724\" (UniqueName: \"kubernetes.io/projected/4cf15517-6d78-429d-8dca-d002ecfc356f-kube-api-access-4f724\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.588064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.587907 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.588130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.588099 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.615001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.614973 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv"] Oct 31 15:06:24.689033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.688995 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f724\" (UniqueName: \"kubernetes.io/projected/4cf15517-6d78-429d-8dca-d002ecfc356f-kube-api-access-4f724\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.689223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.689039 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.689292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.689227 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.689766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.689744 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.692010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.691986 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.748331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.748297 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f724\" (UniqueName: \"kubernetes.io/projected/4cf15517-6d78-429d-8dca-d002ecfc356f-kube-api-access-4f724\") pod \"maas-api-revoke-sub-866444d4-lgvnv\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:24.872006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:24.871936 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:25.264900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.264846 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:25.266111 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:25.266084 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf15517_6d78_429d_8dca_d002ecfc356f.slice/crio-31499bd61e39b48d8e57c46a0831ec18e4a8b2424c2f80e3a2e3d6651b544995 WatchSource:0}: Error finding container 31499bd61e39b48d8e57c46a0831ec18e4a8b2424c2f80e3a2e3d6651b544995: Status 404 returned error can't find the container with id 31499bd61e39b48d8e57c46a0831ec18e4a8b2424c2f80e3a2e3d6651b544995 Oct 31 15:06:25.268634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.268610 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv"] Oct 31 15:06:25.430001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.429965 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" event={"ID":"4cf15517-6d78-429d-8dca-d002ecfc356f","Type":"ContainerStarted","Data":"31499bd61e39b48d8e57c46a0831ec18e4a8b2424c2f80e3a2e3d6651b544995"} Oct 31 15:06:25.556411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.556387 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:25.575576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.575548 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:25.598075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.598024 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-xks-ca\") pod \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " Oct 31 15:06:25.598193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.598133 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa6534b-4e97-4111-8c31-c1bfadee55ac-kube-api-access-7krc9\" (UniqueName: \"kubernetes.io/projected/2aa6534b-4e97-4111-8c31-c1bfadee55ac-kube-api-access-7krc9\") pod \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " Oct 31 15:06:25.598325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.598311 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-openshift-ca\") pod \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\" (UID: \"2aa6534b-4e97-4111-8c31-c1bfadee55ac\") " Oct 31 15:06:25.598684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.598658 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-openshift-ca" pod "2aa6534b-4e97-4111-8c31-c1bfadee55ac" (UID: "2aa6534b-4e97-4111-8c31-c1bfadee55ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:06:25.598930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.598799 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:25.600346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.600322 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-xks-ca" pod "2aa6534b-4e97-4111-8c31-c1bfadee55ac" (UID: "2aa6534b-4e97-4111-8c31-c1bfadee55ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:06:25.600434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.600360 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa6534b-4e97-4111-8c31-c1bfadee55ac-kube-api-access-7krc9" pod "2aa6534b-4e97-4111-8c31-c1bfadee55ac" (UID: "2aa6534b-4e97-4111-8c31-c1bfadee55ac"). InnerVolumeSpecName "kube-api-access-7krc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:25.700827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.700795 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7krc9\" (UniqueName: \"kubernetes.io/projected/2aa6534b-4e97-4111-8c31-c1bfadee55ac-kube-api-access-7krc9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:25.700970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:25.700832 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa6534b-4e97-4111-8c31-c1bfadee55ac-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:26.438215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:26.438177 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" Oct 31 15:06:26.438689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:26.438209 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh" event={"ID":"2aa6534b-4e97-4111-8c31-c1bfadee55ac","Type":"ContainerDied","Data":"619030ed0a1cddd54ff34c8896d76f84394ba99a7a37a63a716590bb22774f8a"} Oct 31 15:06:26.438689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:26.438270 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619030ed0a1cddd54ff34c8896d76f84394ba99a7a37a63a716590bb22774f8a" Oct 31 15:06:26.439929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:26.439896 4865 generic.go:356] "Generic (PLEG): container finished" podID="4cf15517-6d78-429d-8dca-d002ecfc356f" containerID="5955cff87e711947a8b5ed1633adddee7d607e51f2bab4505f1f3422f3a6a803" exitCode=0 Oct 31 15:06:26.440043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:26.439933 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" event={"ID":"4cf15517-6d78-429d-8dca-d002ecfc356f","Type":"ContainerDied","Data":"5955cff87e711947a8b5ed1633adddee7d607e51f2bab4505f1f3422f3a6a803"} Oct 31 15:06:27.049526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.049492 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:27.062133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.062109 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:27.458718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.458690 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:06:27.613051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.613025 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:27.625273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.625249 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-openshift-ca\") pod \"4cf15517-6d78-429d-8dca-d002ecfc356f\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " Oct 31 15:06:27.625438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.625339 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf15517-6d78-429d-8dca-d002ecfc356f-kube-api-access-4f724\" (UniqueName: \"kubernetes.io/projected/4cf15517-6d78-429d-8dca-d002ecfc356f-kube-api-access-4f724\") pod \"4cf15517-6d78-429d-8dca-d002ecfc356f\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " Oct 31 15:06:27.625595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.625574 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-openshift-ca" pod "4cf15517-6d78-429d-8dca-d002ecfc356f" (UID: "4cf15517-6d78-429d-8dca-d002ecfc356f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:06:27.625859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.625843 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:27.627443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.627422 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf15517-6d78-429d-8dca-d002ecfc356f-kube-api-access-4f724" pod "4cf15517-6d78-429d-8dca-d002ecfc356f" (UID: "4cf15517-6d78-429d-8dca-d002ecfc356f"). InnerVolumeSpecName "kube-api-access-4f724". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:27.726974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.726923 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-xks-ca\") pod \"4cf15517-6d78-429d-8dca-d002ecfc356f\" (UID: \"4cf15517-6d78-429d-8dca-d002ecfc356f\") " Oct 31 15:06:27.728183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.728158 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f724\" (UniqueName: \"kubernetes.io/projected/4cf15517-6d78-429d-8dca-d002ecfc356f-kube-api-access-4f724\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:27.729570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.729539 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-xks-ca" pod "4cf15517-6d78-429d-8dca-d002ecfc356f" (UID: "4cf15517-6d78-429d-8dca-d002ecfc356f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:06:27.829372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:27.829342 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf15517-6d78-429d-8dca-d002ecfc356f-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:28.403354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:28.403323 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:28.466246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:28.459598 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" Oct 31 15:06:28.466748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:28.466354 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv" event={"ID":"4cf15517-6d78-429d-8dca-d002ecfc356f","Type":"ContainerDied","Data":"31499bd61e39b48d8e57c46a0831ec18e4a8b2424c2f80e3a2e3d6651b544995"} Oct 31 15:06:28.466748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:28.466452 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31499bd61e39b48d8e57c46a0831ec18e4a8b2424c2f80e3a2e3d6651b544995" Oct 31 15:06:32.786314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.786285 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq"] Oct 31 15:06:32.786847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.786834 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4cf15517-6d78-429d-8dca-d002ecfc356f" containerName="revoke-keys" Oct 31 15:06:32.786891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.786848 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf15517-6d78-429d-8dca-d002ecfc356f" containerName="revoke-keys" Oct 31 15:06:32.786891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.786863 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2aa6534b-4e97-4111-8c31-c1bfadee55ac" containerName="revoke-keys" Oct 31 15:06:32.786891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.786869 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa6534b-4e97-4111-8c31-c1bfadee55ac" containerName="revoke-keys" Oct 31 15:06:32.786992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.786980 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf15517-6d78-429d-8dca-d002ecfc356f" containerName="revoke-keys" Oct 31 15:06:32.786992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.786988 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa6534b-4e97-4111-8c31-c1bfadee55ac" containerName="revoke-keys" Oct 31 15:06:32.790828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.790814 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:32.896708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.896679 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:06:32.896708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.896696 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:06:32.896976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.896684 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:06:32.896976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.896846 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:06:32.897245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.897223 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:32.897323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.897280 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnmn\" (UniqueName: \"kubernetes.io/projected/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-kube-api-access-swnmn\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:32.897498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.897448 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:32.900935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.900917 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq"] Oct 31 15:06:32.998794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.998760 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:32.998981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.998888 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:32.998981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.998934 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnmn\" (UniqueName: \"kubernetes.io/projected/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-kube-api-access-swnmn\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:32.999402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:32.999377 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:33.001462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:33.001439 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:33.062596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:33.062541 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnmn\" (UniqueName: \"kubernetes.io/projected/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-kube-api-access-swnmn\") pod \"maas-api-revoke-sub-617c8ee1-h7mpq\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:33.104273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:33.104249 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:33.334322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:33.334162 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:33.519608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:33.519582 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq"] Oct 31 15:06:33.520022 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:33.519995 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a1b93f_bfff_4ef0_8af5_ffa45849d077.slice/crio-69cb3ca06c28f585e61c8d8564018a34d92cbdefdf9154148db62e52bfdc0b03 WatchSource:0}: Error finding container 69cb3ca06c28f585e61c8d8564018a34d92cbdefdf9154148db62e52bfdc0b03: Status 404 returned error can't find the container with id 69cb3ca06c28f585e61c8d8564018a34d92cbdefdf9154148db62e52bfdc0b03 Oct 31 15:06:34.521521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:34.521459 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" event={"ID":"a4a1b93f-bfff-4ef0-8af5-ffa45849d077","Type":"ContainerStarted","Data":"75b8ad36f1e7aee35cd41056773a28ed2f97c2b5d2861ada21426b9abb598382"} Oct 31 15:06:34.521992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:34.521545 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" event={"ID":"a4a1b93f-bfff-4ef0-8af5-ffa45849d077","Type":"ContainerStarted","Data":"69cb3ca06c28f585e61c8d8564018a34d92cbdefdf9154148db62e52bfdc0b03"} Oct 31 15:06:34.646630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:34.645911 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" podStartSLOduration=2.645890937 podStartE2EDuration="2.645890937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:34.639919171 +0000 UTC m=+1240.259470582" watchObservedRunningTime="2026-10-31 15:06:34.645890937 +0000 UTC m=+1240.265442350" Oct 31 15:06:35.345493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:35.330493 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:35.563495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:35.563393 4865 generic.go:356] "Generic (PLEG): container finished" podID="a4a1b93f-bfff-4ef0-8af5-ffa45849d077" containerID="75b8ad36f1e7aee35cd41056773a28ed2f97c2b5d2861ada21426b9abb598382" exitCode=0 Oct 31 15:06:35.564019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:35.563622 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" event={"ID":"a4a1b93f-bfff-4ef0-8af5-ffa45849d077","Type":"ContainerDied","Data":"75b8ad36f1e7aee35cd41056773a28ed2f97c2b5d2861ada21426b9abb598382"} Oct 31 15:06:37.083481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.083448 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:37.161965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.161926 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-openshift-ca\") pod \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " Oct 31 15:06:37.162903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.162876 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-xks-ca\") pod \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " Oct 31 15:06:37.163412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.163392 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-kube-api-access-swnmn\" (UniqueName: \"kubernetes.io/projected/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-kube-api-access-swnmn\") pod \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\" (UID: \"a4a1b93f-bfff-4ef0-8af5-ffa45849d077\") " Oct 31 15:06:37.164302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.162582 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-openshift-ca" pod "a4a1b93f-bfff-4ef0-8af5-ffa45849d077" (UID: "a4a1b93f-bfff-4ef0-8af5-ffa45849d077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:06:37.166925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.166893 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-kube-api-access-swnmn" pod "a4a1b93f-bfff-4ef0-8af5-ffa45849d077" (UID: "a4a1b93f-bfff-4ef0-8af5-ffa45849d077"). InnerVolumeSpecName "kube-api-access-swnmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:37.173039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.172977 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-xks-ca" pod "a4a1b93f-bfff-4ef0-8af5-ffa45849d077" (UID: "a4a1b93f-bfff-4ef0-8af5-ffa45849d077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:06:37.266779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.266638 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:37.266779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.266680 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swnmn\" (UniqueName: \"kubernetes.io/projected/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-kube-api-access-swnmn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:37.266779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.266697 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a1b93f-bfff-4ef0-8af5-ffa45849d077-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:37.429601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.429560 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:06:37.429900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.429868 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:37.486593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.486560 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-6hmfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:06:37.486754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.486608 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:37.587671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.587575 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" event={"ID":"a4a1b93f-bfff-4ef0-8af5-ffa45849d077","Type":"ContainerDied","Data":"69cb3ca06c28f585e61c8d8564018a34d92cbdefdf9154148db62e52bfdc0b03"} Oct 31 15:06:37.587671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.587640 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69cb3ca06c28f585e61c8d8564018a34d92cbdefdf9154148db62e52bfdc0b03" Oct 31 15:06:37.587933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:37.587710 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq" Oct 31 15:06:38.551179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:38.551148 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:41.575653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:41.575608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:43.238127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:43.238093 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:46.069024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.068988 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:46.430279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.430196 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l"] Oct 31 15:06:46.430765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.430752 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a4a1b93f-bfff-4ef0-8af5-ffa45849d077" containerName="revoke-keys" Oct 31 15:06:46.430811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.430767 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a1b93f-bfff-4ef0-8af5-ffa45849d077" containerName="revoke-keys" Oct 31 15:06:46.430894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.430886 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a1b93f-bfff-4ef0-8af5-ffa45849d077" containerName="revoke-keys" Oct 31 15:06:46.434324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.434309 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.486308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.486264 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.487088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.486773 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.487088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.486935 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw27p\" (UniqueName: \"kubernetes.io/projected/53f1c39b-e088-496e-91bf-7abd6bab96e9-kube-api-access-pw27p\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.514665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.514629 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l"] Oct 31 15:06:46.539957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.539929 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:06:46.539957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.539948 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:06:46.540157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.539969 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:06:46.540157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.540019 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:06:46.587815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.587786 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.587971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.587937 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw27p\" (UniqueName: \"kubernetes.io/projected/53f1c39b-e088-496e-91bf-7abd6bab96e9-kube-api-access-pw27p\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.588016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.587987 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.588707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.588662 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.590442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.590421 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.603648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.603626 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn"] Oct 31 15:06:46.608658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.608636 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.664554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.664518 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn"] Oct 31 15:06:46.733885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.730081 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5695787b-openshift-default-dockercfg-zpbf4\"" Oct 31 15:06:46.737376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.737344 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:46.747520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.747492 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw27p\" (UniqueName: \"kubernetes.io/projected/53f1c39b-e088-496e-91bf-7abd6bab96e9-kube-api-access-pw27p\") pod \"maas-api-revoke-sub-e30cb11a-d5p6l\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.759949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.759922 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848373 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848495 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848567 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848605 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848645 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848777 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848843 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.848931 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.849527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.849066 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzv4\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-kube-api-access-bhzv4\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.883319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.883284 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.950985 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951051 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951099 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951132 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951157 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951257 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951306 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951341 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.951538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.951432 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzv4\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-kube-api-access-bhzv4\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.954000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.953629 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.954816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.954793 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.955196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.955176 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.955383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.955357 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.955709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.955686 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.956071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.956049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:46.957252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:46.957208 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:47.035540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.031354 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzv4\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-kube-api-access-bhzv4\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:47.035540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.034221 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:47.064488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.063027 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:47.249998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.249967 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:47.250414 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:47.250387 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f1c39b_e088_496e_91bf_7abd6bab96e9.slice/crio-e3595a986d3cf41d67fe3ad29f21adaeb81a92c7448463c6945ee34856cca757 WatchSource:0}: Error finding container e3595a986d3cf41d67fe3ad29f21adaeb81a92c7448463c6945ee34856cca757: Status 404 returned error can't find the container with id e3595a986d3cf41d67fe3ad29f21adaeb81a92c7448463c6945ee34856cca757 Oct 31 15:06:47.250672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.250655 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l"] Oct 31 15:06:47.490418 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:06:47.490380 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:06:47.682987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.682958 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn"] Oct 31 15:06:47.683158 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:47.683139 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fce4fd_989c_4828_b307_12bdd3c0cdd3.slice/crio-9366f5c6c8367bf1346512e3fc4c6b7e9c4ccc19b79116c461566a1996a6061d WatchSource:0}: Error finding container 9366f5c6c8367bf1346512e3fc4c6b7e9c4ccc19b79116c461566a1996a6061d: Status 404 returned error can't find the container with id 9366f5c6c8367bf1346512e3fc4c6b7e9c4ccc19b79116c461566a1996a6061d Oct 31 15:06:47.685671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.685637 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:47.685757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.685706 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:47.685757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.685735 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:47.826204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.826164 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" event={"ID":"07fce4fd-989c-4828-b307-12bdd3c0cdd3","Type":"ContainerStarted","Data":"6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23"} Oct 31 15:06:47.826492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.826437 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" event={"ID":"07fce4fd-989c-4828-b307-12bdd3c0cdd3","Type":"ContainerStarted","Data":"9366f5c6c8367bf1346512e3fc4c6b7e9c4ccc19b79116c461566a1996a6061d"} Oct 31 15:06:47.828740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.828717 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" event={"ID":"53f1c39b-e088-496e-91bf-7abd6bab96e9","Type":"ContainerStarted","Data":"7b5e2178959d0e626d20af9668d3482d988920531f2cd047b72865d1bb4b78e3"} Oct 31 15:06:47.828864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.828756 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" event={"ID":"53f1c39b-e088-496e-91bf-7abd6bab96e9","Type":"ContainerStarted","Data":"e3595a986d3cf41d67fe3ad29f21adaeb81a92c7448463c6945ee34856cca757"} Oct 31 15:06:47.948875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:47.948786 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podStartSLOduration=1.948771883 podStartE2EDuration="1.948771883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:47.944280813 +0000 UTC m=+1253.563832455" watchObservedRunningTime="2026-10-31 15:06:47.948771883 +0000 UTC m=+1253.568323288" Oct 31 15:06:48.064920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.064869 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" podStartSLOduration=2.064855654 podStartE2EDuration="2.064855654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:48.062340694 +0000 UTC m=+1253.681892105" watchObservedRunningTime="2026-10-31 15:06:48.064855654 +0000 UTC m=+1253.684407064" Oct 31 15:06:48.098589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.098560 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:48.251048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.251010 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:48.257432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.257407 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:48.840535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.840500 4865 generic.go:356] "Generic (PLEG): container finished" podID="53f1c39b-e088-496e-91bf-7abd6bab96e9" containerID="7b5e2178959d0e626d20af9668d3482d988920531f2cd047b72865d1bb4b78e3" exitCode=0 Oct 31 15:06:48.840710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.840574 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" event={"ID":"53f1c39b-e088-496e-91bf-7abd6bab96e9","Type":"ContainerDied","Data":"7b5e2178959d0e626d20af9668d3482d988920531f2cd047b72865d1bb4b78e3"} Oct 31 15:06:48.841693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.841671 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:48.842660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:48.842641 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:06:50.007097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.007073 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:50.110405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.110318 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-xks-ca\") pod \"53f1c39b-e088-496e-91bf-7abd6bab96e9\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " Oct 31 15:06:50.110615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.110416 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-openshift-ca\") pod \"53f1c39b-e088-496e-91bf-7abd6bab96e9\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " Oct 31 15:06:50.110615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.110517 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f1c39b-e088-496e-91bf-7abd6bab96e9-kube-api-access-pw27p\" (UniqueName: \"kubernetes.io/projected/53f1c39b-e088-496e-91bf-7abd6bab96e9-kube-api-access-pw27p\") pod \"53f1c39b-e088-496e-91bf-7abd6bab96e9\" (UID: \"53f1c39b-e088-496e-91bf-7abd6bab96e9\") " Oct 31 15:06:50.110828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.110804 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-openshift-ca" pod "53f1c39b-e088-496e-91bf-7abd6bab96e9" (UID: "53f1c39b-e088-496e-91bf-7abd6bab96e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:06:50.112741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.112712 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-xks-ca" pod "53f1c39b-e088-496e-91bf-7abd6bab96e9" (UID: "53f1c39b-e088-496e-91bf-7abd6bab96e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:06:50.112848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.112745 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f1c39b-e088-496e-91bf-7abd6bab96e9-kube-api-access-pw27p" pod "53f1c39b-e088-496e-91bf-7abd6bab96e9" (UID: "53f1c39b-e088-496e-91bf-7abd6bab96e9"). InnerVolumeSpecName "kube-api-access-pw27p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:06:50.212609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.212574 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:50.212609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.212606 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f1c39b-e088-496e-91bf-7abd6bab96e9-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:50.212609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.212616 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw27p\" (UniqueName: \"kubernetes.io/projected/53f1c39b-e088-496e-91bf-7abd6bab96e9-kube-api-access-pw27p\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:06:50.857764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.857682 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" Oct 31 15:06:50.857764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.857706 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l" event={"ID":"53f1c39b-e088-496e-91bf-7abd6bab96e9","Type":"ContainerDied","Data":"e3595a986d3cf41d67fe3ad29f21adaeb81a92c7448463c6945ee34856cca757"} Oct 31 15:06:50.858012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.857776 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3595a986d3cf41d67fe3ad29f21adaeb81a92c7448463c6945ee34856cca757" Oct 31 15:06:50.920081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:50.920047 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:51.260490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:51.260449 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:51.402028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:51.401996 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:51.647375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:51.647293 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:52.832372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:52.832338 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:52.957096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:52.957057 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:53.086347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.086266 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:53.559437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.559405 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p"] Oct 31 15:06:53.560018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.560003 4865 cpu_manager.go:414] "containerMap: removing container" podUID="53f1c39b-e088-496e-91bf-7abd6bab96e9" containerName="revoke-keys" Oct 31 15:06:53.560061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.560020 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f1c39b-e088-496e-91bf-7abd6bab96e9" containerName="revoke-keys" Oct 31 15:06:53.560150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.560141 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f1c39b-e088-496e-91bf-7abd6bab96e9" containerName="revoke-keys" Oct 31 15:06:53.563913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.563894 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.610765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.610741 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd"] Oct 31 15:06:53.614751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.614736 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.620379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.620361 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-56bc96-openshift-default-dockercfg-5qxbd\"" Oct 31 15:06:53.663221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663195 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.663355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663227 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.663355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663251 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.663355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663331 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4hb\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-kube-api-access-ss4hb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.663504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663370 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.663504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663392 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9jb\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-kube-api-access-8z9jb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.663504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663486 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.663605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663528 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.663605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663532 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p"] Oct 31 15:06:53.663605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663588 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.663704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663645 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.663704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663679 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.663777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663755 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.663896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663882 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.663932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663921 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.663966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.663947 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9113d241-1dd7-4d9f-a2af-c8043456e64b-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.664033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.664010 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10516171-e5bf-4736-a327-3b00e55df65e-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.664069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.664056 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10516171-e5bf-4736-a327-3b00e55df65e-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.664111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.664075 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.677849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.677821 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd"] Oct 31 15:06:53.765225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765189 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4hb\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-kube-api-access-ss4hb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.765225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765226 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.765499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765244 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9jb\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-kube-api-access-8z9jb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.765499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765271 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.765499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765299 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.765499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765328 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.765499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765401 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.765499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765446 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.765811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765535 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.765811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765688 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.765811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765746 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.765811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765778 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9113d241-1dd7-4d9f-a2af-c8043456e64b-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765808 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10516171-e5bf-4736-a327-3b00e55df65e-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765853 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765874 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765861 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10516171-e5bf-4736-a327-3b00e55df65e-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765878 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765903 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765921 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.765982 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.766010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.766005 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.766464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.766032 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.766464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.766054 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.766464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.766443 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.767151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.766707 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.767151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.766801 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10516171-e5bf-4736-a327-3b00e55df65e-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.767391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.767274 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9113d241-1dd7-4d9f-a2af-c8043456e64b-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.767391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.766454 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.768523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.768501 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.768659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.768555 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.768710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.768686 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10516171-e5bf-4736-a327-3b00e55df65e-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.768927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.768912 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.824303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.824233 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.825206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.825173 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9jb\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-kube-api-access-8z9jb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:53.825427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.825404 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4hb\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-kube-api-access-ss4hb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.825611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.825589 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.878099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.878068 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:53.927740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:53.927708 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:54.161842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.161777 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:54.370518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.364926 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:54.390782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.384534 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:54.390782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.384638 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:54.390782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.384685 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:54.390782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.390616 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p"] Oct 31 15:06:54.404731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.400348 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd"] Oct 31 15:06:54.433507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.432215 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:54.433964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.433933 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:54.434143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.434123 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:06:54.602973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.602946 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb"] Oct 31 15:06:54.611485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.611450 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:54.643989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.643936 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb"] Oct 31 15:06:54.682172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.681932 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5695787b-dockercfg-twqxz\"" Oct 31 15:06:54.682172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.682025 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5695787b\"" Oct 31 15:06:54.844730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.844695 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8877cb4f-44d1-456e-a403-0ff085dfdcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:54.848354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.848327 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qlk\" (UniqueName: \"kubernetes.io/projected/8877cb4f-44d1-456e-a403-0ff085dfdcfd-kube-api-access-49qlk\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:54.958561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.955166 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8877cb4f-44d1-456e-a403-0ff085dfdcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:54.965425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.965233 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8877cb4f-44d1-456e-a403-0ff085dfdcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:54.976488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.974747 4865 scope.go:122] "RemoveContainer" containerID="8105a5a0480bf72d170dfe6d6472f5172421fb72ce3f7f84bc6153fc43fd68b4" Oct 31 15:06:54.976488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.975497 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qlk\" (UniqueName: \"kubernetes.io/projected/8877cb4f-44d1-456e-a403-0ff085dfdcfd-kube-api-access-49qlk\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:54.985224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.981375 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd"] Oct 31 15:06:54.987176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.987157 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:54.992513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.992491 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd"] Oct 31 15:06:55.000750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.999890 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" event={"ID":"9113d241-1dd7-4d9f-a2af-c8043456e64b","Type":"ContainerStarted","Data":"08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522"} Oct 31 15:06:55.000750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:54.999944 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" event={"ID":"9113d241-1dd7-4d9f-a2af-c8043456e64b","Type":"ContainerStarted","Data":"40f9f8d6fb503c8b78ac0e8aab1caa8fed3a2ab84b61090801119876c5d32df2"} Oct 31 15:06:55.002656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.002634 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" event={"ID":"10516171-e5bf-4736-a327-3b00e55df65e","Type":"ContainerStarted","Data":"6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c"} Oct 31 15:06:55.002826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.002805 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" event={"ID":"10516171-e5bf-4736-a327-3b00e55df65e","Type":"ContainerStarted","Data":"694f39bb8fa2f4567d5f8b52aee2047a7de25a443e330c162be0e4de61acf268"} Oct 31 15:06:55.003318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.003297 4865 scope.go:122] "RemoveContainer" containerID="5b5d16c6bf048ef41c7a57ee50dafe73686791ea3758df40d7569be1f46327d1" Oct 31 15:06:55.020164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.020142 4865 scope.go:122] "RemoveContainer" containerID="ad0d7fc78a1c8e66937f4af6d6d1fb2111f5fe01743f0406f11ed1e200324085" Oct 31 15:06:55.034514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.034488 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qlk\" (UniqueName: \"kubernetes.io/projected/8877cb4f-44d1-456e-a403-0ff085dfdcfd-kube-api-access-49qlk\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:55.077037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.076998 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db35cd3-437b-4a02-ae0b-ac45aa570895-plugins-config-volume\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:55.077187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.077048 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4prd\" (UniqueName: \"kubernetes.io/projected/1db35cd3-437b-4a02-ae0b-ac45aa570895-kube-api-access-x4prd\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:55.178996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.178961 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db35cd3-437b-4a02-ae0b-ac45aa570895-plugins-config-volume\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:55.179232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.179015 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4prd\" (UniqueName: \"kubernetes.io/projected/1db35cd3-437b-4a02-ae0b-ac45aa570895-kube-api-access-x4prd\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:55.179972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.179943 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db35cd3-437b-4a02-ae0b-ac45aa570895-plugins-config-volume\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:55.226329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.226228 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podStartSLOduration=2.226208121 podStartE2EDuration="2.226208121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:55.221375936 +0000 UTC m=+1260.840927347" watchObservedRunningTime="2026-10-31 15:06:55.226208121 +0000 UTC m=+1260.845759512" Oct 31 15:06:55.232113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.232081 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:55.237204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.237179 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4prd\" (UniqueName: \"kubernetes.io/projected/1db35cd3-437b-4a02-ae0b-ac45aa570895-kube-api-access-x4prd\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:55.313275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.313248 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:55.350499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.348794 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podStartSLOduration=2.3487694 podStartE2EDuration="2.3487694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:55.341727662 +0000 UTC m=+1260.961279078" watchObservedRunningTime="2026-10-31 15:06:55.3487694 +0000 UTC m=+1260.968320812" Oct 31 15:06:55.621547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.621345 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb"] Oct 31 15:06:55.747600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.747571 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd"] Oct 31 15:06:55.749954 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:06:55.749926 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db35cd3_437b_4a02_ae0b_ac45aa570895.slice/crio-ac52a0cdee8dc5e29c6f26df99738f4a1727b2106796127f41e9017cb909d96f WatchSource:0}: Error finding container ac52a0cdee8dc5e29c6f26df99738f4a1727b2106796127f41e9017cb909d96f: Status 404 returned error can't find the container with id ac52a0cdee8dc5e29c6f26df99738f4a1727b2106796127f41e9017cb909d96f Oct 31 15:06:55.878778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.878755 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:06:55.880370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.880346 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:06:55.880455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.880394 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:06:55.942493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.942089 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:06:55.949753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.949603 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:06:55.949753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:55.949670 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:06:56.051007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.050970 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" event={"ID":"8877cb4f-44d1-456e-a403-0ff085dfdcfd","Type":"ContainerStarted","Data":"a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55"} Oct 31 15:06:56.051388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.051033 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" event={"ID":"8877cb4f-44d1-456e-a403-0ff085dfdcfd","Type":"ContainerStarted","Data":"71d5d886f62af3b7ea1631af15ffdd12fe1107b410c53251a175f36dd00492bb"} Oct 31 15:06:56.051388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.051298 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:06:56.053030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.053000 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" event={"ID":"1db35cd3-437b-4a02-ae0b-ac45aa570895","Type":"ContainerStarted","Data":"bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5"} Oct 31 15:06:56.053168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.053038 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" event={"ID":"1db35cd3-437b-4a02-ae0b-ac45aa570895","Type":"ContainerStarted","Data":"ac52a0cdee8dc5e29c6f26df99738f4a1727b2106796127f41e9017cb909d96f"} Oct 31 15:06:56.053252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.053229 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:06:56.181908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.181808 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" podStartSLOduration=2.181793169 podStartE2EDuration="2.181793169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:56.178086432 +0000 UTC m=+1261.797637843" watchObservedRunningTime="2026-10-31 15:06:56.181793169 +0000 UTC m=+1261.801344579" Oct 31 15:06:56.337495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.329257 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" podStartSLOduration=2.32923469 podStartE2EDuration="2.32923469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:56.311295223 +0000 UTC m=+1261.930846636" watchObservedRunningTime="2026-10-31 15:06:56.32923469 +0000 UTC m=+1261.948786102" Oct 31 15:06:56.650720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.650675 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:06:56.878971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.878940 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:06:56.879140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.878991 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:06:56.936550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.932331 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:06:56.936550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:56.932396 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:06:57.892497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:57.881834 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:06:57.892497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:57.881896 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:06:57.930494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:57.930392 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:06:57.930494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:57.930450 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:06:58.271507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.270989 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:58.654666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.654553 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:58.696956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.696923 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:58.735557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.734588 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:58.821636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.821588 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:58.823312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.823268 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:06:58.856617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.856567 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:58.882629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.882593 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:06:58.882915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.882887 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:06:58.930973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.930883 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:06:58.931565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.931530 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:06:58.945733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.945696 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:58.979516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:58.977568 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:06:59.164605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.164556 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:59.300612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.300568 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:59.352497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.351576 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:06:59.450459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.450420 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp"] Oct 31 15:06:59.450864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.450742 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:59.475088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.475004 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.530859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.530750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:06:59.616340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.615821 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c3356b-openshift-default-dockercfg-rm5sf\"" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.655898 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-certs\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.655959 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-podinfo\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.656189 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-credential-socket\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.656258 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-envoy\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.656318 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-token\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.656348 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9k4\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-kube-api-access-wn9k4\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.656398 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-socket\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.659522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.656420 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-data\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.689529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.685295 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:59.705368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.690032 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:06:59.736548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.735607 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:59.736548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.736205 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:06:59.840578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.823759 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:59.840578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.824108 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:06:59.840578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.824774 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:06:59.840578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.824834 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:06:59.840578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.828611 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:59.873595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.869912 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:06:59.873595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.870323 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:06:59.887979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.886830 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp"] Oct 31 15:06:59.887979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.887311 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:06:59.887979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.887359 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:06:59.935539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.901235 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istiod-ca-cert\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:06:59.935539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.932606 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:06:59.935539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.932664 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:06:59.991493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:06:59.990890 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.018078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.018039 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-credential-socket\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.039495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.018405 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-envoy\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.039798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.039766 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-token\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.039936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.039919 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9k4\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-kube-api-access-wn9k4\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.040120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.040102 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-socket\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.058670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.058634 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-data\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.208150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.199707 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-socket\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.208150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.037669 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:07:00.208150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.042047 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-credential-socket\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.208150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.164492 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-envoy\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.208150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.165322 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:07:00.228786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.098842 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-data\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.238147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.237859 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:07:00.317497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.258117 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9k4\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-kube-api-access-wn9k4\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.343586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.199779 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istiod-ca-cert\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.343965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.343942 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-certs\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.344114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.344098 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-podinfo\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.366497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.310597 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.397027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.367061 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-certs\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.397027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.380173 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:07:00.415176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.317793 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istiod-ca-cert\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.415176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.415016 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Oct 31 15:07:00.432546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.416979 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.452094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.451946 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:07:00.468563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.458157 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:00.468563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.461296 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.494494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.491560 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:07:00.548062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.547970 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.695978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.656633 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-podinfo\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.700656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.700622 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.701349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.701322 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-token\") pod \"e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.835554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.828886 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.858574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.857117 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:07:00.859925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.859899 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:00.914843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.914721 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:00.915021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.914828 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:00.965093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.965044 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.037513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:00.978851 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:01.037513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.019143 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:01.037513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.006612 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:07:01.231758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.231702 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:07:01.373024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.372982 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.405494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.402917 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:07:01.405494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.402979 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.491560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.429245 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.578495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.466558 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:07:01.636496 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:07:01.574956 4865 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.082s" Oct 31 15:07:01.810317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.808763 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:07:01.810317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.725750 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:07:01.875653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.874651 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.877828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.877798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:01.902341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.902292 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:01.991546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.904395 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-2fbkj" podUID="1fd003d5-b7f4-42bf-87ef-bfa1a035d906" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 15:07:01.992253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.916887 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.992680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.916784 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.994294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.991564 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:01.994546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.960502 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:01.998539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:01.995603 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:02.030344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.030296 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:07:02.037556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.030754 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.197483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.197312 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:02.302494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.287709 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.302494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.287803 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:07:02.380652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.380608 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.589460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.589416 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.680501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.629964 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.811596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.811552 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.881685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.881578 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.903508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.900599 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:02.903508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.900658 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:02.936988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.936956 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:02.937250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.937213 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:02.996445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.994553 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.996445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:02.994995 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:07:03.006605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.006565 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:07:03.038620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.038573 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.039808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.039774 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.136565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.124030 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp"] Oct 31 15:07:03.320498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.288719 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.337496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.326577 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:03.341496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.337772 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:03.341764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.341728 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:03.382717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.382671 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.665595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.665556 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:07:03.669586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.669533 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.680968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.680880 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.830437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.827762 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:07:03.882571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.882320 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.921489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.921266 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:07:03.921489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.921348 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.940817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.938612 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:03.940817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.938672 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:03.954711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.954577 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:03.954711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.954642 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:04.002494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:03.998083 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.027378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.014222 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.059497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.048609 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.059497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.049442 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.123495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.120777 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" event={"ID":"967ee42d-2dd6-4a02-9ad1-9f67ddae877b","Type":"ContainerStarted","Data":"90376822487af31258ed50ae2dc354a0242d2da502b1f3b090c1669993348e48"} Oct 31 15:07:04.311009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.310959 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.396495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.395585 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:07:04.695550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.695224 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.699495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.696377 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.781233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.781185 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.848492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.847063 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.906492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.893663 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:07:04.906492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.894922 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:04.906492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.894966 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:04.942496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.940980 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:04.942496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:04.941057 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:05.010504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.009640 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:07:05.017603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.017549 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:07:05.051126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.051080 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.082495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.052627 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.257529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.250607 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.313655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.313554 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.548060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.536522 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.722597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.696657 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:07:05.722597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.698091 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.730505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.729826 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:05.799919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.786068 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" event={"ID":"967ee42d-2dd6-4a02-9ad1-9f67ddae877b","Type":"ContainerStarted","Data":"aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303"} Oct 31 15:07:05.883492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.882938 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:05.883492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.883087 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.908495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.902594 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:05.908495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.902656 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:05.908495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.902765 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:05.937128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:05.937071 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:06.157546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.156766 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:06.157546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.156841 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:06.162817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.162785 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:07:06.163324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.163290 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:07:06.181742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.181705 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:07:06.183196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.183168 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:07:06.312373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.312339 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:07:06.366556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.346834 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" podStartSLOduration=7.346813535 podStartE2EDuration="7.346813535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:06.238188032 +0000 UTC m=+1271.857739444" watchObservedRunningTime="2026-10-31 15:07:06.346813535 +0000 UTC m=+1271.966364946" Oct 31 15:07:06.548523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.543326 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:06.788159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.763357 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:07:06.788159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.763725 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:07:06.838515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.838351 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": context deadline exceeded" Oct 31 15:07:06.895123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.893606 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:06.916696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.916569 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:06.934806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.933621 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:06.934806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.933690 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:06.936166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.936055 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:06.936166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:06.936105 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:07.127127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.127050 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:07.163012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.147011 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:07:07.163012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.156708 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:07:07.163822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.163782 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:07.164312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.164287 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:07:07.204707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.204658 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:07.205326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.205295 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:07:07.256547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.244007 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:07:07.315604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.315554 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:07.451776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.451682 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:07:07.452046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.452013 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:07.556508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.552595 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:07.772735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.772679 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:07:07.773375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.773338 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:07:07.896365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.896329 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:07.928900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.916511 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 31 15:07:07.929206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.929178 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 31 15:07:07.929326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.917214 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:07:07.929764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.917176 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:07:07.952869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.939941 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:07.952869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:07.940004 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:08.074551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.069370 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:08.172749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.172704 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:08.173698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.173665 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:08.228493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.227727 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:08.228493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.228057 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:07:08.287931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.279175 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:07:08.287931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.285450 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:07:08.287931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.286687 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:07:08.287931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.287083 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:07:08.321643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.320890 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:07:08.322553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.322534 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:07:08.322686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.322627 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:07:08.327242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.327187 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:07:08.331543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.331523 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:07:08.878568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.878235 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:07:08.928546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.928512 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:07:08.929333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.929297 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 15:07:08.929569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.929410 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 15:07:08.981009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:08.980974 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:09.182809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:09.182727 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:09.337152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:09.337118 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:09.441773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:09.441642 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:07:09.443899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:09.443876 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:07:09.933123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:09.933096 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:07:09.934225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:09.934208 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:07:11.042849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:11.042794 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:12.204345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.204306 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:12.401492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.394722 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:12.646491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.642785 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9"] Oct 31 15:07:12.657492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.653527 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:12.784493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.736242 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9"] Oct 31 15:07:12.847061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.749802 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c3356b\"" Oct 31 15:07:12.850536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.749910 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c3356b-dockercfg-fqjzd\"" Oct 31 15:07:12.866884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.826663 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbcv\" (UniqueName: \"kubernetes.io/projected/74f6f824-4ec9-4c5c-9372-3a316820801f-kube-api-access-tkbcv\") pod \"payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:12.867208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.867182 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6f824-4ec9-4c5c-9372-3a316820801f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:12.867719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.857754 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9"] Oct 31 15:07:12.956913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.956839 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:12.960032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.959606 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9"] Oct 31 15:07:12.972021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.971990 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbcv\" (UniqueName: \"kubernetes.io/projected/74f6f824-4ec9-4c5c-9372-3a316820801f-kube-api-access-tkbcv\") pod \"payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:12.972154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.972049 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6f824-4ec9-4c5c-9372-3a316820801f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:12.972873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:12.972854 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6f824-4ec9-4c5c-9372-3a316820801f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:13.031342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.031303 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbcv\" (UniqueName: \"kubernetes.io/projected/74f6f824-4ec9-4c5c-9372-3a316820801f-kube-api-access-tkbcv\") pod \"payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:13.073439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.073408 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a01dcf2-1da5-41d6-a045-7061a9d6c004-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:13.073652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.073620 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9kj\" (UniqueName: \"kubernetes.io/projected/9a01dcf2-1da5-41d6-a045-7061a9d6c004-kube-api-access-gq9kj\") pod \"payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:13.175377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.175341 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a01dcf2-1da5-41d6-a045-7061a9d6c004-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:13.175582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.175436 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9kj\" (UniqueName: \"kubernetes.io/projected/9a01dcf2-1da5-41d6-a045-7061a9d6c004-kube-api-access-gq9kj\") pod \"payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:13.176067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.176049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a01dcf2-1da5-41d6-a045-7061a9d6c004-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:13.234083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.234044 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9kj\" (UniqueName: \"kubernetes.io/projected/9a01dcf2-1da5-41d6-a045-7061a9d6c004-kube-api-access-gq9kj\") pod \"payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:13.279640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.279609 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:13.282450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.282425 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:13.660946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.660868 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9"] Oct 31 15:07:13.700198 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:07:13.700175 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a01dcf2_1da5_41d6_a045_7061a9d6c004.slice/crio-669e202a3b0332cdf3cdb72e79adef275322f9b5fe32121f0fdd78135d6dad10 WatchSource:0}: Error finding container 669e202a3b0332cdf3cdb72e79adef275322f9b5fe32121f0fdd78135d6dad10: Status 404 returned error can't find the container with id 669e202a3b0332cdf3cdb72e79adef275322f9b5fe32121f0fdd78135d6dad10 Oct 31 15:07:13.712614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:13.712591 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9"] Oct 31 15:07:13.713688 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:07:13.713665 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f6f824_4ec9_4c5c_9372_3a316820801f.slice/crio-5e57d210bf78b10d57730c4135e2fb1f3d84b5235364aa98cb447cb7f9eb0025 WatchSource:0}: Error finding container 5e57d210bf78b10d57730c4135e2fb1f3d84b5235364aa98cb447cb7f9eb0025: Status 404 returned error can't find the container with id 5e57d210bf78b10d57730c4135e2fb1f3d84b5235364aa98cb447cb7f9eb0025 Oct 31 15:07:14.228131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.228100 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" event={"ID":"9a01dcf2-1da5-41d6-a045-7061a9d6c004","Type":"ContainerStarted","Data":"218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520"} Oct 31 15:07:14.228330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.228181 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" event={"ID":"9a01dcf2-1da5-41d6-a045-7061a9d6c004","Type":"ContainerStarted","Data":"669e202a3b0332cdf3cdb72e79adef275322f9b5fe32121f0fdd78135d6dad10"} Oct 31 15:07:14.228330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.228205 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:14.229807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.229778 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" event={"ID":"74f6f824-4ec9-4c5c-9372-3a316820801f","Type":"ContainerStarted","Data":"4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844"} Oct 31 15:07:14.229937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.229823 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" event={"ID":"74f6f824-4ec9-4c5c-9372-3a316820801f","Type":"ContainerStarted","Data":"5e57d210bf78b10d57730c4135e2fb1f3d84b5235364aa98cb447cb7f9eb0025"} Oct 31 15:07:14.229937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.229848 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:14.345672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.345610 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" podStartSLOduration=2.345593125 podStartE2EDuration="2.345593125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:14.344400023 +0000 UTC m=+1279.963951434" watchObservedRunningTime="2026-10-31 15:07:14.345593125 +0000 UTC m=+1279.965144536" Oct 31 15:07:14.481884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:14.479083 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" podStartSLOduration=2.479011411 podStartE2EDuration="2.479011411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:14.477903808 +0000 UTC m=+1280.097455220" watchObservedRunningTime="2026-10-31 15:07:14.479011411 +0000 UTC m=+1280.098562826" Oct 31 15:07:16.000975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:16.000945 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:17.490762 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:07:17.490716 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:07:20.992628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:20.991604 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:25.240804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:25.240771 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:07:25.241290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:25.241158 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:07:25.748506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:25.741591 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:25.887497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:25.887432 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:29.611839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.611807 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-54ht9-z6pww"] Oct 31 15:07:29.617664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.617642 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.668752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.668725 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-54ht9-z6pww"] Oct 31 15:07:29.709713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.709680 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-service-ca-bundle\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.709882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.709718 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p57w\" (UniqueName: \"kubernetes.io/projected/92f8f994-c9d3-4bcf-99d8-6344c8942685-kube-api-access-6p57w\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.709882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.709747 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/92f8f994-c9d3-4bcf-99d8-6344c8942685-archives-path\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.709990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.709963 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-trusted-ca-bundle\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.810921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.810891 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-service-ca-bundle\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.810921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.810925 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p57w\" (UniqueName: \"kubernetes.io/projected/92f8f994-c9d3-4bcf-99d8-6344c8942685-kube-api-access-6p57w\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.811178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.810946 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/92f8f994-c9d3-4bcf-99d8-6344c8942685-archives-path\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.811178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.811034 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-trusted-ca-bundle\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.811435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.811408 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/92f8f994-c9d3-4bcf-99d8-6344c8942685-archives-path\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.811663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.811639 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-service-ca-bundle\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.811762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.811744 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-trusted-ca-bundle\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.871517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.871444 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p57w\" (UniqueName: \"kubernetes.io/projected/92f8f994-c9d3-4bcf-99d8-6344c8942685-kube-api-access-6p57w\") pod \"periodic-gathering-54ht9-z6pww\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:29.931455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:29.931425 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:07:30.308153 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:07:30.308125 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f8f994_c9d3_4bcf_99d8_6344c8942685.slice/crio-0aebe9f7df13c40c12e0c77da5f94a394964b7b13060128826dc015e1503277b WatchSource:0}: Error finding container 0aebe9f7df13c40c12e0c77da5f94a394964b7b13060128826dc015e1503277b: Status 404 returned error can't find the container with id 0aebe9f7df13c40c12e0c77da5f94a394964b7b13060128826dc015e1503277b Oct 31 15:07:30.309226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:30.309203 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-54ht9-z6pww"] Oct 31 15:07:30.401638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:30.401606 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54ht9-z6pww" event={"ID":"92f8f994-c9d3-4bcf-99d8-6344c8942685","Type":"ContainerStarted","Data":"0f6795452392b86a52c0d2c25dc0a55e459e7eb9be7ffe6abad2157d8286a165"} Oct 31 15:07:30.401795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:30.401652 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54ht9-z6pww" event={"ID":"92f8f994-c9d3-4bcf-99d8-6344c8942685","Type":"ContainerStarted","Data":"0aebe9f7df13c40c12e0c77da5f94a394964b7b13060128826dc015e1503277b"} Oct 31 15:07:30.524022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:30.523972 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-54ht9-z6pww" podStartSLOduration=1.523956713 podStartE2EDuration="1.523956713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:30.52160267 +0000 UTC m=+1296.141154103" watchObservedRunningTime="2026-10-31 15:07:30.523956713 +0000 UTC m=+1296.143508124" Oct 31 15:07:30.564850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:30.564774 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:35.498877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:35.498779 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:37.425218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:37.425184 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:07:38.952639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:38.952607 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh"] Oct 31 15:07:38.957100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:38.957083 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045152 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045189 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045386 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045463 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744f6109-6a92-428a-9567-a268d095ca81-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045630 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045663 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045692 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045846 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48dp\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-kube-api-access-c48dp\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.045932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.045896 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744f6109-6a92-428a-9567-a268d095ca81-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147328 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147362 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147382 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147429 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48dp\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-kube-api-access-c48dp\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147457 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744f6109-6a92-428a-9567-a268d095ca81-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147582 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147618 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147676 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.147842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147699 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744f6109-6a92-428a-9567-a268d095ca81-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.148006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147881 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.148006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.147853 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.148122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.148088 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.148267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.148155 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.148336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.148262 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744f6109-6a92-428a-9567-a268d095ca81-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.149868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.149837 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.150174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.150156 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744f6109-6a92-428a-9567-a268d095ca81-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.174075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.174046 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh"] Oct 31 15:07:39.222011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.221947 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.223153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.223131 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48dp\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-kube-api-access-c48dp\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.271529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.271499 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:39.741463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.741430 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh"] Oct 31 15:07:39.741760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.741711 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:39.741871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.741853 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:39.741954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:39.741939 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:40.476494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:40.473879 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:40.626282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:40.626240 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" event={"ID":"744f6109-6a92-428a-9567-a268d095ca81","Type":"ContainerStarted","Data":"aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492"} Oct 31 15:07:40.626616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:40.626592 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" event={"ID":"744f6109-6a92-428a-9567-a268d095ca81","Type":"ContainerStarted","Data":"95a220deda932cce757bf533ae5874e7b467357bbbf33ab381a43f1d77225123"} Oct 31 15:07:40.861604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:40.861491 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podStartSLOduration=2.861456849 podStartE2EDuration="2.861456849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:40.806933689 +0000 UTC m=+1306.426485136" watchObservedRunningTime="2026-10-31 15:07:40.861456849 +0000 UTC m=+1306.481008259" Oct 31 15:07:41.272189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:41.272163 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:41.273153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:41.273129 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 15:07:41.273341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:41.273322 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 15:07:42.272222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:42.272192 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 15:07:42.272691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:42.272549 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 15:07:43.272241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:43.272206 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 15:07:43.272771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:43.272258 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 15:07:44.272486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:44.272447 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 15:07:44.272859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:44.272532 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 15:07:45.272378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:45.272346 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 15:07:45.272589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:45.272411 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 15:07:45.392522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:45.392419 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:45.532836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:45.532756 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:45.703491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:45.693741 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:46.272583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:46.272548 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 15:07:46.272984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:46.272628 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 15:07:47.495559 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:07:47.495445 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:07:47.752220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:47.752186 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:47.752839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:47.752812 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:47.753877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:47.753853 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:07:48.763359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:48.763327 4865 generic.go:356] "Generic (PLEG): container finished" podID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerID="6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23" exitCode=0 Oct 31 15:07:48.763842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:48.763400 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" event={"ID":"07fce4fd-989c-4828-b307-12bdd3c0cdd3","Type":"ContainerDied","Data":"6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23"} Oct 31 15:07:48.764012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:48.763995 4865 scope.go:122] "RemoveContainer" containerID="6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23" Oct 31 15:07:48.765398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:48.765373 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:48.765450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:48.765437 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:48.765514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:48.765501 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:07:48.840839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:48.840811 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:07:49.772737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:49.772703 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" event={"ID":"07fce4fd-989c-4828-b307-12bdd3c0cdd3","Type":"ContainerStarted","Data":"6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81"} Oct 31 15:07:50.250262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:50.250221 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:07:50.251661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:50.251638 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:07:50.251729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:50.251693 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 15:07:50.255747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:50.255725 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:51.250986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:51.250946 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:07:51.251383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:51.251019 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 15:07:52.013495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:52.010909 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:52.250650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:52.250615 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:07:52.250836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:52.250674 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 15:07:53.251025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:53.250994 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:07:53.251421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:53.251053 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 15:07:53.411233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:53.411180 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:54.251062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:54.251028 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:07:54.251598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:54.251237 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 15:07:55.038913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:55.038875 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:07:55.066261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:55.066236 4865 scope.go:122] "RemoveContainer" containerID="56744f5409c65f05bce581e59e1af344dec12f5f07bf8420fdaefb0bb470e4e8" Oct 31 15:07:55.254677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:55.254646 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:07:55.255115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:55.255103 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:07:55.255807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:55.255788 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:07:59.818629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:07:59.818592 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:02.032889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:02.032854 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:08:04.547362 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:04.547332 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:06.224275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.224239 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8"] Oct 31 15:08:06.228647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.228626 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.265093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.265073 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.265238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.265216 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6lf\" (UniqueName: \"kubernetes.io/projected/8baaa68f-1047-46ce-9d45-7536bad3074c-kube-api-access-2l6lf\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.265371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.265353 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.285382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.285357 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8"] Oct 31 15:08:06.331646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.331607 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:08:06.331819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.331649 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:08:06.331819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.331688 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:08:06.333935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.333914 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:08:06.366579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.366548 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.366738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.366662 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.366805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.366751 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6lf\" (UniqueName: \"kubernetes.io/projected/8baaa68f-1047-46ce-9d45-7536bad3074c-kube-api-access-2l6lf\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.367337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.367316 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.369155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.369133 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.431236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.431203 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6lf\" (UniqueName: \"kubernetes.io/projected/8baaa68f-1047-46ce-9d45-7536bad3074c-kube-api-access-2l6lf\") pod \"maas-api-revoke-sub-2d617cbf-r55c8\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.541797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.541719 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:06.945155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:06.945126 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8"] Oct 31 15:08:06.958058 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:08:06.957826 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baaa68f_1047_46ce_9d45_7536bad3074c.slice/crio-2245a8c40bca2ff04ca0df8eede9354d85175d200894d08d206fa26d289b1761 WatchSource:0}: Error finding container 2245a8c40bca2ff04ca0df8eede9354d85175d200894d08d206fa26d289b1761: Status 404 returned error can't find the container with id 2245a8c40bca2ff04ca0df8eede9354d85175d200894d08d206fa26d289b1761 Oct 31 15:08:07.035140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:07.035104 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" event={"ID":"8baaa68f-1047-46ce-9d45-7536bad3074c","Type":"ContainerStarted","Data":"2245a8c40bca2ff04ca0df8eede9354d85175d200894d08d206fa26d289b1761"} Oct 31 15:08:08.046217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:08.046173 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" event={"ID":"8baaa68f-1047-46ce-9d45-7536bad3074c","Type":"ContainerStarted","Data":"f465cb222ec02f6d4a2ac396cd546c7b88d7a007ac95f638147e0dcb55017876"} Oct 31 15:08:08.179154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:08.179104 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" podStartSLOduration=2.179091022 podStartE2EDuration="2.179091022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:08.176094712 +0000 UTC m=+1333.795646123" watchObservedRunningTime="2026-10-31 15:08:08.179091022 +0000 UTC m=+1333.798642433" Oct 31 15:08:09.057400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:09.057369 4865 generic.go:356] "Generic (PLEG): container finished" podID="8baaa68f-1047-46ce-9d45-7536bad3074c" containerID="f465cb222ec02f6d4a2ac396cd546c7b88d7a007ac95f638147e0dcb55017876" exitCode=0 Oct 31 15:08:09.057824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:09.057459 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" event={"ID":"8baaa68f-1047-46ce-9d45-7536bad3074c","Type":"ContainerDied","Data":"f465cb222ec02f6d4a2ac396cd546c7b88d7a007ac95f638147e0dcb55017876"} Oct 31 15:08:09.343603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:09.343519 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:10.253069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.253041 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:10.364428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.364335 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-xks-ca\") pod \"8baaa68f-1047-46ce-9d45-7536bad3074c\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " Oct 31 15:08:10.364659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.364452 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-openshift-ca\") pod \"8baaa68f-1047-46ce-9d45-7536bad3074c\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " Oct 31 15:08:10.364659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.364564 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8baaa68f-1047-46ce-9d45-7536bad3074c-kube-api-access-2l6lf\" (UniqueName: \"kubernetes.io/projected/8baaa68f-1047-46ce-9d45-7536bad3074c-kube-api-access-2l6lf\") pod \"8baaa68f-1047-46ce-9d45-7536bad3074c\" (UID: \"8baaa68f-1047-46ce-9d45-7536bad3074c\") " Oct 31 15:08:10.364861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.364836 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-openshift-ca" pod "8baaa68f-1047-46ce-9d45-7536bad3074c" (UID: "8baaa68f-1047-46ce-9d45-7536bad3074c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:08:10.365164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.365145 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:10.367346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.367320 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-xks-ca" pod "8baaa68f-1047-46ce-9d45-7536bad3074c" (UID: "8baaa68f-1047-46ce-9d45-7536bad3074c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:08:10.367433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.367351 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baaa68f-1047-46ce-9d45-7536bad3074c-kube-api-access-2l6lf" pod "8baaa68f-1047-46ce-9d45-7536bad3074c" (UID: "8baaa68f-1047-46ce-9d45-7536bad3074c"). InnerVolumeSpecName "kube-api-access-2l6lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:08:10.466675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.466638 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8baaa68f-1047-46ce-9d45-7536bad3074c-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:10.466675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:10.466672 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l6lf\" (UniqueName: \"kubernetes.io/projected/8baaa68f-1047-46ce-9d45-7536bad3074c-kube-api-access-2l6lf\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:11.075867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:11.075841 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" Oct 31 15:08:11.076074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:11.075873 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8" event={"ID":"8baaa68f-1047-46ce-9d45-7536bad3074c","Type":"ContainerDied","Data":"2245a8c40bca2ff04ca0df8eede9354d85175d200894d08d206fa26d289b1761"} Oct 31 15:08:11.076074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:11.075928 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2245a8c40bca2ff04ca0df8eede9354d85175d200894d08d206fa26d289b1761" Oct 31 15:08:12.352428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:12.352354 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:13.094897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:13.094867 4865 generic.go:356] "Generic (PLEG): container finished" podID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerID="aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303" exitCode=0 Oct 31 15:08:13.095080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:13.094946 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" event={"ID":"967ee42d-2dd6-4a02-9ad1-9f67ddae877b","Type":"ContainerDied","Data":"aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303"} Oct 31 15:08:13.095376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:13.095362 4865 scope.go:122] "RemoveContainer" containerID="aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303" Oct 31 15:08:13.096933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:13.096900 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:13.097055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:13.096986 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:13.097055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:13.097034 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:14.123034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:14.123002 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" event={"ID":"967ee42d-2dd6-4a02-9ad1-9f67ddae877b","Type":"ContainerStarted","Data":"13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203"} Oct 31 15:08:14.516374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:14.516339 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:14.879161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:14.879075 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:08:14.879342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:14.879207 4865 patch_prober.go:29] interesting pod/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 15:08:14.879342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:14.879276 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 15:08:15.879849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:15.879815 4865 patch_prober.go:29] interesting pod/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 15:08:15.880245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:15.879869 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 15:08:16.083252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.083221 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm"] Oct 31 15:08:16.083839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.083824 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8baaa68f-1047-46ce-9d45-7536bad3074c" containerName="revoke-keys" Oct 31 15:08:16.083888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.083841 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8baaa68f-1047-46ce-9d45-7536bad3074c" containerName="revoke-keys" Oct 31 15:08:16.083970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.083961 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8baaa68f-1047-46ce-9d45-7536bad3074c" containerName="revoke-keys" Oct 31 15:08:16.087337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.087321 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.144645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.144565 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm"] Oct 31 15:08:16.167657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.167630 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.167799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.167685 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w69tw\" (UniqueName: \"kubernetes.io/projected/a9ccf549-27f3-4945-941b-32cce06cc473-kube-api-access-w69tw\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.167799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.167777 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.190764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.190738 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:08:16.190934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.190790 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:08:16.191606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.191587 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:08:16.191681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.191593 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:08:16.268979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.268941 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.269176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.269023 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69tw\" (UniqueName: \"kubernetes.io/projected/a9ccf549-27f3-4945-941b-32cce06cc473-kube-api-access-w69tw\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.269176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.269060 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.269766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.269746 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.271480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.271443 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.327126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.327095 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w69tw\" (UniqueName: \"kubernetes.io/projected/a9ccf549-27f3-4945-941b-32cce06cc473-kube-api-access-w69tw\") pod \"maas-api-revoke-sub-54130add-nk9zm\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.423639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.423547 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:16.424897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.424873 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:16.842498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.834595 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:16.900495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.896915 4865 patch_prober.go:29] interesting pod/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 15:08:16.900495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:16.896979 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 15:08:17.088018 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:08:17.087990 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ccf549_27f3_4945_941b_32cce06cc473.slice/crio-ab8a6fdae9b33b69979d4b54e2bd57946b3d5f8749bf630bd411bbb7928814d5 WatchSource:0}: Error finding container ab8a6fdae9b33b69979d4b54e2bd57946b3d5f8749bf630bd411bbb7928814d5: Status 404 returned error can't find the container with id ab8a6fdae9b33b69979d4b54e2bd57946b3d5f8749bf630bd411bbb7928814d5 Oct 31 15:08:17.090091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:17.090016 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm"] Oct 31 15:08:17.145028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:17.144997 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" event={"ID":"a9ccf549-27f3-4945-941b-32cce06cc473","Type":"ContainerStarted","Data":"ab8a6fdae9b33b69979d4b54e2bd57946b3d5f8749bf630bd411bbb7928814d5"} Oct 31 15:08:17.491045 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:08:17.491011 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:08:17.621664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:17.621630 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:17.879698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:17.879621 4865 patch_prober.go:29] interesting pod/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 15:08:17.879698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:17.879669 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 15:08:18.130739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.130661 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw"] Oct 31 15:08:18.135337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.135312 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.173726 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-credential-socket\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.173838 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-envoy\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.173870 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-certs\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.173931 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-data\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.173990 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-podinfo\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.174015 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istiod-ca-cert\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.174068 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-socket\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.174233 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-token\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.174522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.174273 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjkrg\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-kube-api-access-hjkrg\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.176651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.175797 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" event={"ID":"a9ccf549-27f3-4945-941b-32cce06cc473","Type":"ContainerStarted","Data":"ef6842911ca75cdaf602d42c03e8719461acd1f1f25abe9f7b41de6bc9d6cf6d"} Oct 31 15:08:18.238232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.227439 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:18.281550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.274740 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4d3ea8-openshift-default-dockercfg-gwd25\"" Oct 31 15:08:18.343563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.278582 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw"] Oct 31 15:08:18.357595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.284192 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-data\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.392683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.284597 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-data\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.392683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.384726 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-podinfo\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.392683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.384777 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istiod-ca-cert\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.392683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.384922 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-socket\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.392683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.391548 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istiod-ca-cert\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.484351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.395023 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-podinfo\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.484351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.395279 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-socket\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.529493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.521069 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:18.691035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.687576 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-token\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.691035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.687646 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjkrg\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-kube-api-access-hjkrg\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.691035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.687723 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-credential-socket\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.691035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.687854 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-envoy\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.691035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.687885 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-certs\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.691035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.689065 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-credential-socket\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.691035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.689230 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-certs\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.738680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.738602 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" podStartSLOduration=2.738579968 podStartE2EDuration="2.738579968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:18.443781368 +0000 UTC m=+1344.063332782" watchObservedRunningTime="2026-10-31 15:08:18.738579968 +0000 UTC m=+1344.358131372" Oct 31 15:08:18.742841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.742723 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-envoy\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.779523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.768732 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-token\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.779523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.768942 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjkrg\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-kube-api-access-hjkrg\") pod \"e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.815495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.811751 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:18.879519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.879486 4865 patch_prober.go:29] interesting pod/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 15:08:18.879725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:18.879544 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 15:08:19.184765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.184731 4865 generic.go:356] "Generic (PLEG): container finished" podID="a9ccf549-27f3-4945-941b-32cce06cc473" containerID="ef6842911ca75cdaf602d42c03e8719461acd1f1f25abe9f7b41de6bc9d6cf6d" exitCode=0 Oct 31 15:08:19.185194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.184829 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" event={"ID":"a9ccf549-27f3-4945-941b-32cce06cc473","Type":"ContainerDied","Data":"ef6842911ca75cdaf602d42c03e8719461acd1f1f25abe9f7b41de6bc9d6cf6d"} Oct 31 15:08:19.219270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.219234 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:19.343640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.343598 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw"] Oct 31 15:08:19.396426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.384876 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:19.396426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.384988 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:19.396426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.385045 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:19.883805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.883778 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:08:19.884292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.884270 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:08:19.884838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:19.884818 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:08:20.197459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.197380 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" event={"ID":"2e890f84-dc95-41f4-9b65-7ce6b635e5a6","Type":"ContainerStarted","Data":"d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337"} Oct 31 15:08:20.197459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.197443 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" event={"ID":"2e890f84-dc95-41f4-9b65-7ce6b635e5a6","Type":"ContainerStarted","Data":"54dc999db8e88df1b8bf555b0aaf04a6ad633d4942a139b04c46423210156bc3"} Oct 31 15:08:20.322545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.322498 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" podStartSLOduration=2.322482976 podStartE2EDuration="2.322482976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:20.318910081 +0000 UTC m=+1345.938461492" watchObservedRunningTime="2026-10-31 15:08:20.322482976 +0000 UTC m=+1345.942034381" Oct 31 15:08:20.365786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.365755 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:20.476490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.469020 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-xks-ca\") pod \"a9ccf549-27f3-4945-941b-32cce06cc473\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " Oct 31 15:08:20.476490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.469160 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ccf549-27f3-4945-941b-32cce06cc473-kube-api-access-w69tw\" (UniqueName: \"kubernetes.io/projected/a9ccf549-27f3-4945-941b-32cce06cc473-kube-api-access-w69tw\") pod \"a9ccf549-27f3-4945-941b-32cce06cc473\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " Oct 31 15:08:20.476490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.469253 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-openshift-ca\") pod \"a9ccf549-27f3-4945-941b-32cce06cc473\" (UID: \"a9ccf549-27f3-4945-941b-32cce06cc473\") " Oct 31 15:08:20.481955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.481760 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-openshift-ca" pod "a9ccf549-27f3-4945-941b-32cce06cc473" (UID: "a9ccf549-27f3-4945-941b-32cce06cc473"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:08:20.483249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.483222 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:20.491115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.486137 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ccf549-27f3-4945-941b-32cce06cc473-kube-api-access-w69tw" pod "a9ccf549-27f3-4945-941b-32cce06cc473" (UID: "a9ccf549-27f3-4945-941b-32cce06cc473"). InnerVolumeSpecName "kube-api-access-w69tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:08:20.521877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.521736 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-xks-ca" pod "a9ccf549-27f3-4945-941b-32cce06cc473" (UID: "a9ccf549-27f3-4945-941b-32cce06cc473"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:08:20.586544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.586512 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ccf549-27f3-4945-941b-32cce06cc473-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:20.586544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.586538 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w69tw\" (UniqueName: \"kubernetes.io/projected/a9ccf549-27f3-4945-941b-32cce06cc473-kube-api-access-w69tw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:20.817572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.817482 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:20.822579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:20.822555 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:21.206295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:21.206220 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" Oct 31 15:08:21.206295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:21.206241 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm" event={"ID":"a9ccf549-27f3-4945-941b-32cce06cc473","Type":"ContainerDied","Data":"ab8a6fdae9b33b69979d4b54e2bd57946b3d5f8749bf630bd411bbb7928814d5"} Oct 31 15:08:21.206295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:21.206291 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8a6fdae9b33b69979d4b54e2bd57946b3d5f8749bf630bd411bbb7928814d5" Oct 31 15:08:21.206823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:21.206715 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:21.207367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:21.207348 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:08:22.775784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:22.775738 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:23.090341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:23.090245 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:23.267806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:23.267774 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:23.415001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:23.414930 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:24.136377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:24.136336 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:24.271049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:24.271009 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:26.307139 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:26.307110 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:26.516116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:26.516082 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:26.928496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:26.924689 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc"] Oct 31 15:08:26.928496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:26.925840 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a9ccf549-27f3-4945-941b-32cce06cc473" containerName="revoke-keys" Oct 31 15:08:26.928496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:26.925880 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ccf549-27f3-4945-941b-32cce06cc473" containerName="revoke-keys" Oct 31 15:08:26.928496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:26.926179 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ccf549-27f3-4945-941b-32cce06cc473" containerName="revoke-keys" Oct 31 15:08:27.092490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.087677 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc"] Oct 31 15:08:27.092490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.087808 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.145317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.145292 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:08:27.145317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.145309 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:08:27.145584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.145506 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:08:27.145584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.145513 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:08:27.176706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.176679 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.176858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.176780 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8dc\" (UniqueName: \"kubernetes.io/projected/cb30a083-1a19-4d31-8189-2d57471e14b5-kube-api-access-gq8dc\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.176910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.176858 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.200781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.200705 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:08:27.278639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.278604 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.278856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.278646 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8dc\" (UniqueName: \"kubernetes.io/projected/cb30a083-1a19-4d31-8189-2d57471e14b5-kube-api-access-gq8dc\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.278856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.278688 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.279371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.279349 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.281226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.281202 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.336703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.336671 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8dc\" (UniqueName: \"kubernetes.io/projected/cb30a083-1a19-4d31-8189-2d57471e14b5-kube-api-access-gq8dc\") pod \"maas-api-revoke-sub-13b48eec-6lwkc\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.418198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.418155 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:27.829543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:27.828414 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc"] Oct 31 15:08:28.351493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:28.350654 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" event={"ID":"cb30a083-1a19-4d31-8189-2d57471e14b5","Type":"ContainerStarted","Data":"5dbcf9dc43adde550ea493e10ec78ed2a73a4d8ec9ac10afe11e5bee8a9bed14"} Oct 31 15:08:28.828507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:28.826578 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:28.955042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:28.954982 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:29.450112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:29.450022 4865 generic.go:356] "Generic (PLEG): container finished" podID="cb30a083-1a19-4d31-8189-2d57471e14b5" containerID="7ae5f0db27c8df75e84b6baefd9084e7dfe578e1d6c13ffe06d76c5cc9d6dbfb" exitCode=0 Oct 31 15:08:29.450854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:29.450827 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" event={"ID":"cb30a083-1a19-4d31-8189-2d57471e14b5","Type":"ContainerDied","Data":"7ae5f0db27c8df75e84b6baefd9084e7dfe578e1d6c13ffe06d76c5cc9d6dbfb"} Oct 31 15:08:29.719603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:29.699628 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.024134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.024024 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.715497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.698775 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 31 15:08:30.715497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.699857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:30.718979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.718943 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.720212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.720183 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:08:30.790423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.665140 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:08:30.819873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.819789 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.849496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.829309 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:08:30.872501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.852749 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.872501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.868836 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.910553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:30.905991 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:31.045106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.045005 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:08:31.060026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.047581 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.061660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.061636 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:08:31.073599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.061839 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.237687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.236830 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.293957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.293893 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:31.736669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.736080 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.750522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.737779 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.753457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.750781 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:08:31.857503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.808581 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.857503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.808958 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:08:31.857503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.822893 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.857503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.824275 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:08:31.959400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.959360 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.964988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:31.964938 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:08:32.154397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.154089 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:08:32.178011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.177974 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:32.218522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.178818 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:08:32.273528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.256049 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:32.273528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.265559 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:32.273528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.265870 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:08:32.339630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.308582 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:32.339630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.308910 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:08:32.486672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.484335 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:32.607064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.597366 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:08:32.607414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.607393 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:08:32.612500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.610263 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:08:32.618286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.618267 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:08:32.655176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.655151 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:08:32.830027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.823077 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:08:32.839934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.838570 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:32.853884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.841923 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:08:32.853884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.842837 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:32.895688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:32.895658 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:33.067588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.061985 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb30a083-1a19-4d31-8189-2d57471e14b5-kube-api-access-gq8dc\" (UniqueName: \"kubernetes.io/projected/cb30a083-1a19-4d31-8189-2d57471e14b5-kube-api-access-gq8dc\") pod \"cb30a083-1a19-4d31-8189-2d57471e14b5\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " Oct 31 15:08:33.075279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.075251 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-openshift-ca\") pod \"cb30a083-1a19-4d31-8189-2d57471e14b5\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " Oct 31 15:08:33.088822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.075769 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-xks-ca\") pod \"cb30a083-1a19-4d31-8189-2d57471e14b5\" (UID: \"cb30a083-1a19-4d31-8189-2d57471e14b5\") " Oct 31 15:08:33.125356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.117695 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-openshift-ca" pod "cb30a083-1a19-4d31-8189-2d57471e14b5" (UID: "cb30a083-1a19-4d31-8189-2d57471e14b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:08:33.129120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.129064 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:33.185497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.170263 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:33.208095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.208055 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-xks-ca" pod "cb30a083-1a19-4d31-8189-2d57471e14b5" (UID: "cb30a083-1a19-4d31-8189-2d57471e14b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:08:33.316438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.224963 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb30a083-1a19-4d31-8189-2d57471e14b5-kube-api-access-gq8dc" pod "cb30a083-1a19-4d31-8189-2d57471e14b5" (UID: "cb30a083-1a19-4d31-8189-2d57471e14b5"). InnerVolumeSpecName "kube-api-access-gq8dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:08:33.316438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.236636 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb30a083-1a19-4d31-8189-2d57471e14b5-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:33.316438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.236668 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq8dc\" (UniqueName: \"kubernetes.io/projected/cb30a083-1a19-4d31-8189-2d57471e14b5-kube-api-access-gq8dc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:08:33.316438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.280287 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:33.316438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.301369 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" Oct 31 15:08:33.316438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.310989 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc" event={"ID":"cb30a083-1a19-4d31-8189-2d57471e14b5","Type":"ContainerDied","Data":"5dbcf9dc43adde550ea493e10ec78ed2a73a4d8ec9ac10afe11e5bee8a9bed14"} Oct 31 15:08:33.316438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.311048 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dbcf9dc43adde550ea493e10ec78ed2a73a4d8ec9ac10afe11e5bee8a9bed14" Oct 31 15:08:33.380936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.380860 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:08:33.480330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.469626 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:08:33.590740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.590711 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:08:33.924248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.923437 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-8l9gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:08:33.924248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:33.923522 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:37.252174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:37.252142 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:37.468063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:37.468029 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:37.640548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:37.639990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:39.412766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.412734 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq"] Oct 31 15:08:39.413358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.413344 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cb30a083-1a19-4d31-8189-2d57471e14b5" containerName="revoke-keys" Oct 31 15:08:39.413358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.413359 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb30a083-1a19-4d31-8189-2d57471e14b5" containerName="revoke-keys" Oct 31 15:08:39.413505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.413493 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb30a083-1a19-4d31-8189-2d57471e14b5" containerName="revoke-keys" Oct 31 15:08:39.418423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.418403 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.528400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.528363 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq"] Oct 31 15:08:39.542487 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.542441 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e1b9e0-f557-43a6-895b-249e01261062-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.542634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.542519 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.542678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.542650 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.542866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.542843 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.542926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.542900 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e1b9e0-f557-43a6-895b-249e01261062-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.543075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.543052 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.543124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.543099 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.543222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.543204 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.543285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.543255 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvc8\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-kube-api-access-zvvc8\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.644982 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e1b9e0-f557-43a6-895b-249e01261062-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645017 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645053 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645154 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645195 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e1b9e0-f557-43a6-895b-249e01261062-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645276 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645307 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645355 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645395 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvc8\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-kube-api-access-zvvc8\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645519 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645560 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645579 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.645882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.645736 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.646378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.646354 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e1b9e0-f557-43a6-895b-249e01261062-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.647910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.647886 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.648027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.648011 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e1b9e0-f557-43a6-895b-249e01261062-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.704379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.704316 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.704531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.704423 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvc8\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-kube-api-access-zvvc8\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:39.733099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:39.733078 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:40.164392 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:08:40.164348 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e1b9e0_f557_43a6_895b_249e01261062.slice/crio-d897fa4855e01a7e51e16fd4de1119f93f274ed06fd349d9d031db02730eee42 WatchSource:0}: Error finding container d897fa4855e01a7e51e16fd4de1119f93f274ed06fd349d9d031db02730eee42: Status 404 returned error can't find the container with id d897fa4855e01a7e51e16fd4de1119f93f274ed06fd349d9d031db02730eee42 Oct 31 15:08:40.165872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.165851 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq"] Oct 31 15:08:40.166817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.166788 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:40.166878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.166851 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:40.166922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.166885 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:08:40.355372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.355335 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" event={"ID":"a8e1b9e0-f557-43a6-895b-249e01261062","Type":"ContainerStarted","Data":"4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277"} Oct 31 15:08:40.355570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.355400 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" event={"ID":"a8e1b9e0-f557-43a6-895b-249e01261062","Type":"ContainerStarted","Data":"d897fa4855e01a7e51e16fd4de1119f93f274ed06fd349d9d031db02730eee42"} Oct 31 15:08:40.477983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.477933 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podStartSLOduration=1.477918818 podStartE2EDuration="1.477918818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:40.473890564 +0000 UTC m=+1366.093441976" watchObservedRunningTime="2026-10-31 15:08:40.477918818 +0000 UTC m=+1366.097470233" Oct 31 15:08:40.494584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.494557 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:40.761298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.761217 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:40.799541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:40.793002 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:41.503652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.503619 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh"] Oct 31 15:08:41.527731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.527689 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh"] Oct 31 15:08:41.529552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.526840 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:41.585923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.585898 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cf5d51-dockercfg-wz4hn\"" Oct 31 15:08:41.586663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.586645 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cf5d51\"" Oct 31 15:08:41.592267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.592245 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvx4\" (UniqueName: \"kubernetes.io/projected/fa54e817-61df-4128-a183-bf589a13c9b9-kube-api-access-8rvx4\") pod \"payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:41.592685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.592665 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa54e817-61df-4128-a183-bf589a13c9b9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:41.694295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.694257 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvx4\" (UniqueName: \"kubernetes.io/projected/fa54e817-61df-4128-a183-bf589a13c9b9-kube-api-access-8rvx4\") pod \"payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:41.694623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.694594 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa54e817-61df-4128-a183-bf589a13c9b9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:41.695566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.695541 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa54e817-61df-4128-a183-bf589a13c9b9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:41.735136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.735106 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:08:41.735308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.735161 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 15:08:41.754254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.754168 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvx4\" (UniqueName: \"kubernetes.io/projected/fa54e817-61df-4128-a183-bf589a13c9b9-kube-api-access-8rvx4\") pod \"payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:41.849200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:41.849171 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:42.254212 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:08:42.254171 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa54e817_61df_4128_a183_bf589a13c9b9.slice/crio-caae2812558b88c4ae00ff997b813827a07305639f019647860b926b1e8adf7f WatchSource:0}: Error finding container caae2812558b88c4ae00ff997b813827a07305639f019647860b926b1e8adf7f: Status 404 returned error can't find the container with id caae2812558b88c4ae00ff997b813827a07305639f019647860b926b1e8adf7f Oct 31 15:08:42.255561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.255540 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh"] Oct 31 15:08:42.531148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.531055 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" event={"ID":"fa54e817-61df-4128-a183-bf589a13c9b9","Type":"ContainerStarted","Data":"4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0"} Oct 31 15:08:42.531148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.531118 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" event={"ID":"fa54e817-61df-4128-a183-bf589a13c9b9","Type":"ContainerStarted","Data":"caae2812558b88c4ae00ff997b813827a07305639f019647860b926b1e8adf7f"} Oct 31 15:08:42.531148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.531146 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:42.537665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.537642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:42.676497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.675127 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" podStartSLOduration=1.675106454 podStartE2EDuration="1.675106454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:42.650904039 +0000 UTC m=+1368.270455465" watchObservedRunningTime="2026-10-31 15:08:42.675106454 +0000 UTC m=+1368.294657868" Oct 31 15:08:42.745496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.733820 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:08:42.745496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:42.733895 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 15:08:43.734135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:43.734099 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:08:43.734588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:43.734156 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 15:08:44.733803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:44.733768 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:08:44.733992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:44.733819 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 15:08:45.734114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:45.734080 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:08:45.734524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:45.734149 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 15:08:46.734154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:46.734121 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:08:46.734735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:46.734188 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 15:08:47.296618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:47.296564 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:47.498281 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:08:47.498233 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:08:47.733938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:47.733906 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:08:47.734154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:47.733965 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 15:08:48.738626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:48.738593 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:48.739038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:48.738967 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:48.739896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:48.739875 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:08:53.138998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:53.138960 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:53.263076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:53.263036 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:53.562519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:53.559749 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:08:53.690273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:53.690239 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:08:56.921481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:08:56.921428 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:00.048151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:00.048107 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7"] Oct 31 15:09:00.080843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:00.080810 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-accf6735-wg6d7"] Oct 31 15:09:00.498400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:00.498368 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690a9272-7b7b-4bbe-9bc1-22f84b0002d5" path="/var/lib/kubelet/pods/690a9272-7b7b-4bbe-9bc1-22f84b0002d5/volumes" Oct 31 15:09:01.710693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:01.710652 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:06.355078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:06.354979 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:06.525005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:06.524953 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:06.662140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:06.660484 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:06.742649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:06.742618 4865 generic.go:356] "Generic (PLEG): container finished" podID="92f8f994-c9d3-4bcf-99d8-6344c8942685" containerID="0f6795452392b86a52c0d2c25dc0a55e459e7eb9be7ffe6abad2157d8286a165" exitCode=0 Oct 31 15:09:06.742825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:06.742706 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54ht9-z6pww" event={"ID":"92f8f994-c9d3-4bcf-99d8-6344c8942685","Type":"ContainerDied","Data":"0f6795452392b86a52c0d2c25dc0a55e459e7eb9be7ffe6abad2157d8286a165"} Oct 31 15:09:06.764180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:06.764147 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:09:07.923690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:07.923646 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:09:08.092222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.092138 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f8f994-c9d3-4bcf-99d8-6344c8942685-kube-api-access-6p57w\" (UniqueName: \"kubernetes.io/projected/92f8f994-c9d3-4bcf-99d8-6344c8942685-kube-api-access-6p57w\") pod \"92f8f994-c9d3-4bcf-99d8-6344c8942685\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " Oct 31 15:09:08.092406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.092350 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-trusted-ca-bundle\") pod \"92f8f994-c9d3-4bcf-99d8-6344c8942685\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " Oct 31 15:09:08.092519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.092451 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-service-ca-bundle\") pod \"92f8f994-c9d3-4bcf-99d8-6344c8942685\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " Oct 31 15:09:08.092592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.092545 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f8f994-c9d3-4bcf-99d8-6344c8942685-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/92f8f994-c9d3-4bcf-99d8-6344c8942685-archives-path\") pod \"92f8f994-c9d3-4bcf-99d8-6344c8942685\" (UID: \"92f8f994-c9d3-4bcf-99d8-6344c8942685\") " Oct 31 15:09:08.092859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.092829 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-trusted-ca-bundle" pod "92f8f994-c9d3-4bcf-99d8-6344c8942685" (UID: "92f8f994-c9d3-4bcf-99d8-6344c8942685"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:09:08.092953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.092886 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-service-ca-bundle" pod "92f8f994-c9d3-4bcf-99d8-6344c8942685" (UID: "92f8f994-c9d3-4bcf-99d8-6344c8942685"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:09:08.092953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.092937 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f8f994-c9d3-4bcf-99d8-6344c8942685-archives-path" pod "92f8f994-c9d3-4bcf-99d8-6344c8942685" (UID: "92f8f994-c9d3-4bcf-99d8-6344c8942685"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:09:08.093280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.093259 4865 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-trusted-ca-bundle\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:08.093280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.093280 4865 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f8f994-c9d3-4bcf-99d8-6344c8942685-service-ca-bundle\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:08.093431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.093289 4865 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/92f8f994-c9d3-4bcf-99d8-6344c8942685-archives-path\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:08.095014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.094989 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f8f994-c9d3-4bcf-99d8-6344c8942685-kube-api-access-6p57w" pod "92f8f994-c9d3-4bcf-99d8-6344c8942685" (UID: "92f8f994-c9d3-4bcf-99d8-6344c8942685"). InnerVolumeSpecName "kube-api-access-6p57w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:09:08.194896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.194867 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p57w\" (UniqueName: \"kubernetes.io/projected/92f8f994-c9d3-4bcf-99d8-6344c8942685-kube-api-access-6p57w\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:08.760911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.760836 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-54ht9-z6pww" Oct 31 15:09:08.760911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.760847 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-54ht9-z6pww" event={"ID":"92f8f994-c9d3-4bcf-99d8-6344c8942685","Type":"ContainerDied","Data":"0aebe9f7df13c40c12e0c77da5f94a394964b7b13060128826dc015e1503277b"} Oct 31 15:09:08.760911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:08.760891 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aebe9f7df13c40c12e0c77da5f94a394964b7b13060128826dc015e1503277b" Oct 31 15:09:09.116334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.116245 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx"] Oct 31 15:09:09.120573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.120542 4865 cpu_manager.go:414] "containerMap: removing container" podUID="92f8f994-c9d3-4bcf-99d8-6344c8942685" containerName="insights-gathering" Oct 31 15:09:09.123639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.123609 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f8f994-c9d3-4bcf-99d8-6344c8942685" containerName="insights-gathering" Oct 31 15:09:09.124122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.124097 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f8f994-c9d3-4bcf-99d8-6344c8942685" containerName="insights-gathering" Oct 31 15:09:09.129555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.129527 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.184222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.184173 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx"] Oct 31 15:09:09.194502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.193788 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:09:09.194502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.194443 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:09:09.212495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.208712 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:09:09.212495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.208806 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:09:09.237492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.231114 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.237492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.231177 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv989\" (UniqueName: \"kubernetes.io/projected/cd05523f-56c9-49c2-a48f-496f5719e765-kube-api-access-lv989\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.237492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.231482 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.369794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.369708 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.370457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.370430 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.370457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.370502 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv989\" (UniqueName: \"kubernetes.io/projected/cd05523f-56c9-49c2-a48f-496f5719e765-kube-api-access-lv989\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.371386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.371362 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.374154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.374128 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.429195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.429163 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv989\" (UniqueName: \"kubernetes.io/projected/cd05523f-56c9-49c2-a48f-496f5719e765-kube-api-access-lv989\") pod \"maas-api-revoke-sub-93da8bef-d62qx\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.457735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.457710 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:09.832856 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:09:09.832827 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd05523f_56c9_49c2_a48f_496f5719e765.slice/crio-9aba345d1591b1c42eaef25c6484bcfc9b7dfeee962600665c4c7b577f2e83e6 WatchSource:0}: Error finding container 9aba345d1591b1c42eaef25c6484bcfc9b7dfeee962600665c4c7b577f2e83e6: Status 404 returned error can't find the container with id 9aba345d1591b1c42eaef25c6484bcfc9b7dfeee962600665c4c7b577f2e83e6 Oct 31 15:09:09.836135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.836110 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx"] Oct 31 15:09:09.958924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:09.958897 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:10.091909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:10.091816 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:10.777657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:10.777622 4865 generic.go:356] "Generic (PLEG): container finished" podID="cd05523f-56c9-49c2-a48f-496f5719e765" containerID="44f8a09b9f0e8077bdf71261a036ad42c10ba1e689e3e48acf79b2156196abd3" exitCode=0 Oct 31 15:09:10.778134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:10.777709 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" event={"ID":"cd05523f-56c9-49c2-a48f-496f5719e765","Type":"ContainerDied","Data":"44f8a09b9f0e8077bdf71261a036ad42c10ba1e689e3e48acf79b2156196abd3"} Oct 31 15:09:10.778134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:10.777762 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" event={"ID":"cd05523f-56c9-49c2-a48f-496f5719e765","Type":"ContainerStarted","Data":"9aba345d1591b1c42eaef25c6484bcfc9b7dfeee962600665c4c7b577f2e83e6"} Oct 31 15:09:11.293009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:11.292979 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:11.965980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:11.965958 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:12.007122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.007100 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-openshift-ca\") pod \"cd05523f-56c9-49c2-a48f-496f5719e765\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " Oct 31 15:09:12.007243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.007163 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-xks-ca\") pod \"cd05523f-56c9-49c2-a48f-496f5719e765\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " Oct 31 15:09:12.007243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.007216 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd05523f-56c9-49c2-a48f-496f5719e765-kube-api-access-lv989\" (UniqueName: \"kubernetes.io/projected/cd05523f-56c9-49c2-a48f-496f5719e765-kube-api-access-lv989\") pod \"cd05523f-56c9-49c2-a48f-496f5719e765\" (UID: \"cd05523f-56c9-49c2-a48f-496f5719e765\") " Oct 31 15:09:12.007580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.007548 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-openshift-ca" pod "cd05523f-56c9-49c2-a48f-496f5719e765" (UID: "cd05523f-56c9-49c2-a48f-496f5719e765"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:09:12.008000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.007924 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:12.009391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.009367 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd05523f-56c9-49c2-a48f-496f5719e765-kube-api-access-lv989" pod "cd05523f-56c9-49c2-a48f-496f5719e765" (UID: "cd05523f-56c9-49c2-a48f-496f5719e765"). InnerVolumeSpecName "kube-api-access-lv989". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:09:12.009508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.009427 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-xks-ca" pod "cd05523f-56c9-49c2-a48f-496f5719e765" (UID: "cd05523f-56c9-49c2-a48f-496f5719e765"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:09:12.110140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.110060 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd05523f-56c9-49c2-a48f-496f5719e765-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:12.110140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.110096 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv989\" (UniqueName: \"kubernetes.io/projected/cd05523f-56c9-49c2-a48f-496f5719e765-kube-api-access-lv989\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:12.794614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.794526 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" Oct 31 15:09:12.794614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.794529 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx" event={"ID":"cd05523f-56c9-49c2-a48f-496f5719e765","Type":"ContainerDied","Data":"9aba345d1591b1c42eaef25c6484bcfc9b7dfeee962600665c4c7b577f2e83e6"} Oct 31 15:09:12.794858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:12.794622 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aba345d1591b1c42eaef25c6484bcfc9b7dfeee962600665c4c7b577f2e83e6" Oct 31 15:09:15.086028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:15.085989 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt"] Oct 31 15:09:15.136280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:15.136252 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb1c88a6-xlxzt"] Oct 31 15:09:16.001848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:16.001813 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:16.047837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:16.047812 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k"] Oct 31 15:09:16.126496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:16.121295 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4d2ca1-w8h6k"] Oct 31 15:09:16.497146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:16.497113 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73341bea-aa9b-47e4-9b6c-0b34e29b9f47" path="/var/lib/kubelet/pods/73341bea-aa9b-47e4-9b6c-0b34e29b9f47/volumes" Oct 31 15:09:16.497601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:16.497586 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee3f5c5-b61d-44bf-9e78-276b57eaf159" path="/var/lib/kubelet/pods/dee3f5c5-b61d-44bf-9e78-276b57eaf159/volumes" Oct 31 15:09:17.491303 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:09:17.491273 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:09:20.646621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:20.646591 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:20.809291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:20.809206 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:20.948136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:20.944884 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:26.014495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.013789 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:26.177495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.176566 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:09:26.383691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.383586 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:26.599611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.599559 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:09:26.622490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.621630 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:09:26.856497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.839574 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:26.856497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.840441 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:26.856497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.850222 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:26.856497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:26.850508 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:09:27.185553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.179558 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:27.185553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.180324 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:09:27.449953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.447059 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:27.450520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.450493 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:09:27.678496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.639641 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:09:27.678496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.640027 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:09:27.678496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.671846 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:27.680661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.680635 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:09:27.762492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.760943 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:27.859600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.859552 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:27.883791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.883756 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:09:27.993092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:27.992974 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:09:28.064545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:28.064434 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:09:28.081616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:28.081527 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:09:28.084542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:28.084517 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:09:28.088498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:28.087315 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:09:28.088498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:28.088056 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:09:30.922049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:30.922011 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:32.047892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:32.047856 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5"] Oct 31 15:09:32.101087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:32.101058 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d4ea60-t7nw5"] Oct 31 15:09:32.497089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:32.497057 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a0d1d4-d59e-4c69-8920-bd81dc3a851e" path="/var/lib/kubelet/pods/55a0d1d4-d59e-4c69-8920-bd81dc3a851e/volumes" Oct 31 15:09:34.179673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:34.179642 4865 generic.go:356] "Generic (PLEG): container finished" podID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerID="d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337" exitCode=0 Oct 31 15:09:34.180118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:34.179725 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" event={"ID":"2e890f84-dc95-41f4-9b65-7ce6b635e5a6","Type":"ContainerDied","Data":"d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337"} Oct 31 15:09:34.180171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:34.180142 4865 scope.go:122] "RemoveContainer" containerID="d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337" Oct 31 15:09:34.181536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:34.181509 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:09:34.181596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:34.181577 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:09:34.181635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:34.181609 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:09:35.189431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:35.189397 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" event={"ID":"2e890f84-dc95-41f4-9b65-7ce6b635e5a6","Type":"ContainerStarted","Data":"67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156"} Oct 31 15:09:35.583205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:35.583175 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:35.818855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:35.818816 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:09:35.820325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:35.820296 4865 patch_prober.go:29] interesting pod/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 15:09:35.820495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:35.820353 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 15:09:36.218665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:36.218630 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:09:36.817116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:36.817078 4865 patch_prober.go:29] interesting pod/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 15:09:36.817381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:36.817349 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 15:09:37.818846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:37.818811 4865 patch_prober.go:29] interesting pod/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 15:09:37.819291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:37.818890 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 15:09:38.818876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:38.818841 4865 patch_prober.go:29] interesting pod/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 15:09:38.819281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:38.818905 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 15:09:39.819575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:39.819494 4865 patch_prober.go:29] interesting pod/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 15:09:39.820255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:39.819572 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 15:09:40.311907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:40.311867 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:40.821885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:40.821859 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:09:40.822853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:40.822830 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:09:41.049245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:41.049210 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg"] Oct 31 15:09:41.053359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:41.053334 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1156bf32-p6rxg"] Oct 31 15:09:42.499004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:42.498975 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4828559a-e3f6-4b7f-8ecd-e006a97b0bba" path="/var/lib/kubelet/pods/4828559a-e3f6-4b7f-8ecd-e006a97b0bba/volumes" Oct 31 15:09:45.039603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:45.039575 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:45.047820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:45.047798 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt"] Oct 31 15:09:45.132276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:45.132242 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7510bdf-2x2mt"] Oct 31 15:09:46.497435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:46.497400 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a" path="/var/lib/kubelet/pods/a6dbf432-c3d9-4ca3-80c5-a2e88b6f288a/volumes" Oct 31 15:09:47.489171 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:09:47.489133 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:09:47.600275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:47.600207 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:48.207701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.207668 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl"] Oct 31 15:09:48.208305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.208292 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cd05523f-56c9-49c2-a48f-496f5719e765" containerName="revoke-keys" Oct 31 15:09:48.208351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.208307 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd05523f-56c9-49c2-a48f-496f5719e765" containerName="revoke-keys" Oct 31 15:09:48.208436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.208426 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd05523f-56c9-49c2-a48f-496f5719e765" containerName="revoke-keys" Oct 31 15:09:48.212122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.212107 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.272452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.272415 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvfp\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-kube-api-access-cbvfp\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.272738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.272714 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-data\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.272973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.272958 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-podinfo\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.273083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.273071 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e639f49-cb86-4ab5-a18e-efe152020a2a-istiod-ca-cert\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.273187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.273175 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-socket\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.273294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.273281 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-certs\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.273423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.273407 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-token\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.273623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.273590 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-credential-socket\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.274047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.274011 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-envoy\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.298988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.298955 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:48.319854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.319820 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-aac61b-openshift-default-dockercfg-bm8z5\"" Oct 31 15:09:48.405933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.381656 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-envoy\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407257 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvfp\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-kube-api-access-cbvfp\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407312 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-data\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407372 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-podinfo\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407404 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e639f49-cb86-4ab5-a18e-efe152020a2a-istiod-ca-cert\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407444 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-socket\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407540 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-certs\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407604 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-token\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.407639 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-credential-socket\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.408607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.408192 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-credential-socket\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.411086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.409994 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e639f49-cb86-4ab5-a18e-efe152020a2a-istiod-ca-cert\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.411086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.410630 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-data\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.416093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.416068 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-certs\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.416650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.416617 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-socket\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.437567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.437538 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl"] Oct 31 15:09:48.468249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.468140 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-podinfo\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.469044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.468994 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-envoy\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.475277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.475211 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-token\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.508494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.507544 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvfp\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-kube-api-access-cbvfp\") pod \"e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.569136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.568815 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:48.724538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:48.719127 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:49.182092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.182016 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl"] Oct 31 15:09:49.219923 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:09:49.219891 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e639f49_cb86_4ab5_a18e_efe152020a2a.slice/crio-012f037a1041bad6e2ab714614b996eccb0136f6d6ebf026b68783d59407e6c5 WatchSource:0}: Error finding container 012f037a1041bad6e2ab714614b996eccb0136f6d6ebf026b68783d59407e6c5: Status 404 returned error can't find the container with id 012f037a1041bad6e2ab714614b996eccb0136f6d6ebf026b68783d59407e6c5 Oct 31 15:09:49.222433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.222398 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:09:49.222554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.222506 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:09:49.222554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.222540 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:09:49.425668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.425628 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" event={"ID":"0e639f49-cb86-4ab5-a18e-efe152020a2a","Type":"ContainerStarted","Data":"9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba"} Oct 31 15:09:49.425860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.425694 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" event={"ID":"0e639f49-cb86-4ab5-a18e-efe152020a2a","Type":"ContainerStarted","Data":"012f037a1041bad6e2ab714614b996eccb0136f6d6ebf026b68783d59407e6c5"} Oct 31 15:09:49.546881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.546824 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podStartSLOduration=1.546809678 podStartE2EDuration="1.546809678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:09:49.543273478 +0000 UTC m=+1435.162824902" watchObservedRunningTime="2026-10-31 15:09:49.546809678 +0000 UTC m=+1435.166361088" Oct 31 15:09:49.610197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.610155 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:49.895874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:49.895797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:50.615960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:50.615929 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:51.049545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:51.049500 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg"] Oct 31 15:09:51.053691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:51.053670 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-465be251-qs2rg"] Oct 31 15:09:51.443771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:51.443694 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:51.444612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:51.444594 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:09:52.276274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.276238 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:52.447360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.447330 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:52.503776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.503743 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92124fc8-c59a-4d9d-a526-7c31e96804e0" path="/var/lib/kubelet/pods/92124fc8-c59a-4d9d-a526-7c31e96804e0/volumes" Oct 31 15:09:52.584923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.584851 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:52.817198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.817169 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:52.899416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.899345 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4"] Oct 31 15:09:52.904004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.903987 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:52.961347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:52.961316 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4"] Oct 31 15:09:53.005690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.005659 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:09:53.005848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.005753 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:09:53.026623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.026599 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:09:53.026744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.026626 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:09:53.070244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.070221 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wljg\" (UniqueName: \"kubernetes.io/projected/64bf3bce-410f-4027-a485-e2cd386ff4ef-kube-api-access-5wljg\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.070407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.070389 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.070536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.070512 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.172545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.172431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wljg\" (UniqueName: \"kubernetes.io/projected/64bf3bce-410f-4027-a485-e2cd386ff4ef-kube-api-access-5wljg\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.172721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.172603 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.172721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.172676 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.173335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.173305 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.175065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.175049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.230390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.230363 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wljg\" (UniqueName: \"kubernetes.io/projected/64bf3bce-410f-4027-a485-e2cd386ff4ef-kube-api-access-5wljg\") pod \"maas-api-revoke-keys-e2e-mt-5695787b-p9rl4\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.518641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.518600 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:53.895215 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:09:53.895185 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bf3bce_410f_4027_a485_e2cd386ff4ef.slice/crio-39ad2e4d49a25f05c761f80c0d1e0ac43dc847d8b10dc1220c3486dd507143f0 WatchSource:0}: Error finding container 39ad2e4d49a25f05c761f80c0d1e0ac43dc847d8b10dc1220c3486dd507143f0: Status 404 returned error can't find the container with id 39ad2e4d49a25f05c761f80c0d1e0ac43dc847d8b10dc1220c3486dd507143f0 Oct 31 15:09:53.898077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:53.898046 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4"] Oct 31 15:09:54.046733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:54.046698 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:54.048502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:54.048483 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj"] Oct 31 15:09:54.052755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:54.052732 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5957f5ba-nqpqj"] Oct 31 15:09:54.469746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:54.469713 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" event={"ID":"64bf3bce-410f-4027-a485-e2cd386ff4ef","Type":"ContainerStarted","Data":"8a62c8bf33af588ba75491521f594d75b960edc79af31443a4aab8a9e596027e"} Oct 31 15:09:54.469959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:54.469779 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" event={"ID":"64bf3bce-410f-4027-a485-e2cd386ff4ef","Type":"ContainerStarted","Data":"39ad2e4d49a25f05c761f80c0d1e0ac43dc847d8b10dc1220c3486dd507143f0"} Oct 31 15:09:54.500102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:54.500070 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb81f782-5a08-4d0b-b40d-8e8f46f60472" path="/var/lib/kubelet/pods/bb81f782-5a08-4d0b-b40d-8e8f46f60472/volumes" Oct 31 15:09:54.586218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:54.586165 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" podStartSLOduration=2.586151521 podStartE2EDuration="2.586151521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:09:54.584405553 +0000 UTC m=+1440.203956964" watchObservedRunningTime="2026-10-31 15:09:54.586151521 +0000 UTC m=+1440.205702931" Oct 31 15:09:55.135738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:55.135658 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:55.480447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:55.480408 4865 generic.go:356] "Generic (PLEG): container finished" podID="64bf3bce-410f-4027-a485-e2cd386ff4ef" containerID="8a62c8bf33af588ba75491521f594d75b960edc79af31443a4aab8a9e596027e" exitCode=0 Oct 31 15:09:55.480640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:55.480497 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" event={"ID":"64bf3bce-410f-4027-a485-e2cd386ff4ef","Type":"ContainerDied","Data":"8a62c8bf33af588ba75491521f594d75b960edc79af31443a4aab8a9e596027e"} Oct 31 15:09:55.927355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:55.927273 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:56.140505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.132371 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:09:56.346760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.346706 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc"] Oct 31 15:09:56.397501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.393951 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:56.453612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.453578 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc"] Oct 31 15:09:56.472389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.464278 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-aac61b-dockercfg-8wq77\"" Oct 31 15:09:56.472683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.472419 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-aac61b\"" Oct 31 15:09:56.525708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.525669 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:56.526011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.525983 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2kj\" (UniqueName: \"kubernetes.io/projected/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-kube-api-access-gx2kj\") pod \"payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:56.583397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.583342 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq"] Oct 31 15:09:56.636161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.636074 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:56.637994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.637956 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2kj\" (UniqueName: \"kubernetes.io/projected/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-kube-api-access-gx2kj\") pod \"payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:56.638604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.637614 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:56.638759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.636436 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:56.782653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.780704 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq"] Oct 31 15:09:56.808077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.808010 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2kj\" (UniqueName: \"kubernetes.io/projected/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-kube-api-access-gx2kj\") pod \"payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:56.868919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.868885 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77gc\" (UniqueName: \"kubernetes.io/projected/ec856fce-255e-4afb-8a4f-546ae2b68ec7-kube-api-access-m77gc\") pod \"payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:56.869103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.869034 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec856fce-255e-4afb-8a4f-546ae2b68ec7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:56.946118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.946095 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:56.970125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.970095 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec856fce-255e-4afb-8a4f-546ae2b68ec7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:56.970361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.970343 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77gc\" (UniqueName: \"kubernetes.io/projected/ec856fce-255e-4afb-8a4f-546ae2b68ec7-kube-api-access-m77gc\") pod \"payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:56.970870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:56.970848 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec856fce-255e-4afb-8a4f-546ae2b68ec7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:57.028825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.028795 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77gc\" (UniqueName: \"kubernetes.io/projected/ec856fce-255e-4afb-8a4f-546ae2b68ec7-kube-api-access-m77gc\") pod \"payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:57.071667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.071643 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bf3bce-410f-4027-a485-e2cd386ff4ef-kube-api-access-5wljg\" (UniqueName: \"kubernetes.io/projected/64bf3bce-410f-4027-a485-e2cd386ff4ef-kube-api-access-5wljg\") pod \"64bf3bce-410f-4027-a485-e2cd386ff4ef\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " Oct 31 15:09:57.071886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.071851 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-openshift-ca\") pod \"64bf3bce-410f-4027-a485-e2cd386ff4ef\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " Oct 31 15:09:57.072032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.072002 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-xks-ca\") pod \"64bf3bce-410f-4027-a485-e2cd386ff4ef\" (UID: \"64bf3bce-410f-4027-a485-e2cd386ff4ef\") " Oct 31 15:09:57.072399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.072333 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-openshift-ca" pod "64bf3bce-410f-4027-a485-e2cd386ff4ef" (UID: "64bf3bce-410f-4027-a485-e2cd386ff4ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:09:57.072776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.072751 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:57.074587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.074564 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-xks-ca" pod "64bf3bce-410f-4027-a485-e2cd386ff4ef" (UID: "64bf3bce-410f-4027-a485-e2cd386ff4ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:09:57.074745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.074718 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bf3bce-410f-4027-a485-e2cd386ff4ef-kube-api-access-5wljg" pod "64bf3bce-410f-4027-a485-e2cd386ff4ef" (UID: "64bf3bce-410f-4027-a485-e2cd386ff4ef"). InnerVolumeSpecName "kube-api-access-5wljg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:09:57.108083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.108054 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:57.108235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.108218 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:57.178104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.175742 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf3bce-410f-4027-a485-e2cd386ff4ef-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:57.178104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.175774 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wljg\" (UniqueName: \"kubernetes.io/projected/64bf3bce-410f-4027-a485-e2cd386ff4ef-kube-api-access-5wljg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:09:57.521311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.521282 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" Oct 31 15:09:57.521311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.521302 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4" event={"ID":"64bf3bce-410f-4027-a485-e2cd386ff4ef","Type":"ContainerDied","Data":"39ad2e4d49a25f05c761f80c0d1e0ac43dc847d8b10dc1220c3486dd507143f0"} Oct 31 15:09:57.521549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.521344 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ad2e4d49a25f05c761f80c0d1e0ac43dc847d8b10dc1220c3486dd507143f0" Oct 31 15:09:57.595145 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:09:57.595120 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4b0d78_9a74_4fa3_80b9_d22ed2c0a4ba.slice/crio-70cb5f79ef776cb578b3dd8ae4f9344ec846b2b792420cd0e152b3b5058ce492 WatchSource:0}: Error finding container 70cb5f79ef776cb578b3dd8ae4f9344ec846b2b792420cd0e152b3b5058ce492: Status 404 returned error can't find the container with id 70cb5f79ef776cb578b3dd8ae4f9344ec846b2b792420cd0e152b3b5058ce492 Oct 31 15:09:57.596682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.596601 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc"] Oct 31 15:09:57.600867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:57.600847 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq"] Oct 31 15:09:57.635812 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:09:57.635791 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec856fce_255e_4afb_8a4f_546ae2b68ec7.slice/crio-cfb50e266b5657c874ec953b3419927f0c8f8a055517bdd2b017e6cc70d6a31d WatchSource:0}: Error finding container cfb50e266b5657c874ec953b3419927f0c8f8a055517bdd2b017e6cc70d6a31d: Status 404 returned error can't find the container with id cfb50e266b5657c874ec953b3419927f0c8f8a055517bdd2b017e6cc70d6a31d Oct 31 15:09:58.529730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.529687 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" event={"ID":"ec856fce-255e-4afb-8a4f-546ae2b68ec7","Type":"ContainerStarted","Data":"fae65899e447648e45de6706ffe815410c08596981b56fcd35550f339286c3e8"} Oct 31 15:09:58.530204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.529753 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" event={"ID":"ec856fce-255e-4afb-8a4f-546ae2b68ec7","Type":"ContainerStarted","Data":"cfb50e266b5657c874ec953b3419927f0c8f8a055517bdd2b017e6cc70d6a31d"} Oct 31 15:09:58.530204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.529784 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:09:58.531333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.531307 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" event={"ID":"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba","Type":"ContainerStarted","Data":"ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576"} Oct 31 15:09:58.531441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.531351 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" event={"ID":"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba","Type":"ContainerStarted","Data":"70cb5f79ef776cb578b3dd8ae4f9344ec846b2b792420cd0e152b3b5058ce492"} Oct 31 15:09:58.531441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.531396 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:09:58.646410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.646360 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" podStartSLOduration=2.646346651 podStartE2EDuration="2.646346651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:09:58.644325845 +0000 UTC m=+1444.263877269" watchObservedRunningTime="2026-10-31 15:09:58.646346651 +0000 UTC m=+1444.265898082" Oct 31 15:09:58.802563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:09:58.799453 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" podStartSLOduration=2.799431084 podStartE2EDuration="2.799431084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:09:58.761463825 +0000 UTC m=+1444.381015238" watchObservedRunningTime="2026-10-31 15:09:58.799431084 +0000 UTC m=+1444.418982497" Oct 31 15:10:00.050203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:00.050159 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd"] Oct 31 15:10:00.127509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:00.124909 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv"] Oct 31 15:10:00.131230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:00.131200 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdcf77c9-8jzgd"] Oct 31 15:10:00.135502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:00.135452 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-636fc0bc-tfrrv"] Oct 31 15:10:00.205245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:00.205211 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:00.499581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:00.499551 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273" path="/var/lib/kubelet/pods/1a2e92ae-4b52-4b1f-b9f8-8e89f9f44273/volumes" Oct 31 15:10:00.500028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:00.500013 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84cb418e-35ed-4a22-b50b-542a8aeb011c" path="/var/lib/kubelet/pods/84cb418e-35ed-4a22-b50b-542a8aeb011c/volumes" Oct 31 15:10:01.505497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:01.504734 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:10:04.056393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:04.056360 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd"] Oct 31 15:10:04.174636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:04.174603 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb"] Oct 31 15:10:04.409500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:04.409403 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:04.573343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:04.573285 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:05.079653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:05.079619 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:05.457901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:05.457870 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:05.831840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:05.831756 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:06.464493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:06.464081 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:06.783571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:06.780580 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:09.048634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:09.048606 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d"] Oct 31 15:10:09.056686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:09.056656 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" podUID="1db35cd3-437b-4a02-ae0b-ac45aa570895" containerName="payload-processing" containerID="cri-o://bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5" gracePeriod=25 Oct 31 15:10:09.099245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:09.099214 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf39bc0-xs28d"] Oct 31 15:10:09.175617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:09.175584 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" podUID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" containerName="payload-pre-processing" containerID="cri-o://a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55" gracePeriod=25 Oct 31 15:10:09.579870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:09.579843 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:10:09.580068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:09.579906 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:10:10.488791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:10.487555 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:10.544246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:10.544207 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf15f47-91e2-400e-9a8e-4444c0e0b9dd" path="/var/lib/kubelet/pods/2bf15f47-91e2-400e-9a8e-4444c0e0b9dd/volumes" Oct 31 15:10:11.294191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:11.294156 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:11.469438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:11.469404 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:11.607344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:11.607262 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:14.500436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:14.500395 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:14.695526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:14.694023 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:14.938268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:14.938167 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp"] Oct 31 15:10:14.939621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:14.939593 4865 cpu_manager.go:414] "containerMap: removing container" podUID="64bf3bce-410f-4027-a485-e2cd386ff4ef" containerName="revoke-keys" Oct 31 15:10:14.939820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:14.939805 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bf3bce-410f-4027-a485-e2cd386ff4ef" containerName="revoke-keys" Oct 31 15:10:14.940179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:14.940163 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bf3bce-410f-4027-a485-e2cd386ff4ef" containerName="revoke-keys" Oct 31 15:10:14.948768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:14.948746 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.030129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.030084 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5f17a1\"" Oct 31 15:10:15.145508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.141446 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmsbw\" (UniqueName: \"kubernetes.io/projected/18a65fdb-497b-4152-938f-854f35e3b818-kube-api-access-mmsbw\") pod \"payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.145508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.142452 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a65fdb-497b-4152-938f-854f35e3b818-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.193746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.193639 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp"] Oct 31 15:10:15.292974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.246509 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmsbw\" (UniqueName: \"kubernetes.io/projected/18a65fdb-497b-4152-938f-854f35e3b818-kube-api-access-mmsbw\") pod \"payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.292974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.246731 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a65fdb-497b-4152-938f-854f35e3b818-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.292974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.247927 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a65fdb-497b-4152-938f-854f35e3b818-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.292974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.278398 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz"] Oct 31 15:10:15.348864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.348837 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.353217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.353181 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71e955c-d595-435b-9702-9ebf6ef2fc70-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.353792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.353760 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6hq\" (UniqueName: \"kubernetes.io/projected/c71e955c-d595-435b-9702-9ebf6ef2fc70-kube-api-access-9t6hq\") pod \"payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.391261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.391231 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmsbw\" (UniqueName: \"kubernetes.io/projected/18a65fdb-497b-4152-938f-854f35e3b818-kube-api-access-mmsbw\") pod \"payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.406462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.406432 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5f17a1-dockercfg-wmvfj\"" Oct 31 15:10:15.412542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.412435 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz"] Oct 31 15:10:15.468678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.458802 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71e955c-d595-435b-9702-9ebf6ef2fc70-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.468678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.461296 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71e955c-d595-435b-9702-9ebf6ef2fc70-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.468678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.465778 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6hq\" (UniqueName: \"kubernetes.io/projected/c71e955c-d595-435b-9702-9ebf6ef2fc70-kube-api-access-9t6hq\") pod \"payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.491534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.483860 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:15.524926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.524898 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6hq\" (UniqueName: \"kubernetes.io/projected/c71e955c-d595-435b-9702-9ebf6ef2fc70-kube-api-access-9t6hq\") pod \"payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.690238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.690213 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:15.809028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.808842 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:15.903535 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:10:15.903463 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a65fdb_497b_4152_938f_854f35e3b818.slice/crio-3de133d12b9af6d1a9492e3ae469aca2565701f92bdada7340cb86d158accdcf WatchSource:0}: Error finding container 3de133d12b9af6d1a9492e3ae469aca2565701f92bdada7340cb86d158accdcf: Status 404 returned error can't find the container with id 3de133d12b9af6d1a9492e3ae469aca2565701f92bdada7340cb86d158accdcf Oct 31 15:10:15.905272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.905249 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp"] Oct 31 15:10:15.929591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:15.929023 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" event={"ID":"18a65fdb-497b-4152-938f-854f35e3b818","Type":"ContainerStarted","Data":"3de133d12b9af6d1a9492e3ae469aca2565701f92bdada7340cb86d158accdcf"} Oct 31 15:10:16.261394 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:10:16.261364 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71e955c_d595_435b_9702_9ebf6ef2fc70.slice/crio-8eb331e6c5c93b4163658951eb4f0a900f6b5ee1035f24fb0e6823a707e789a3 WatchSource:0}: Error finding container 8eb331e6c5c93b4163658951eb4f0a900f6b5ee1035f24fb0e6823a707e789a3: Status 404 returned error can't find the container with id 8eb331e6c5c93b4163658951eb4f0a900f6b5ee1035f24fb0e6823a707e789a3 Oct 31 15:10:16.263182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:16.263162 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz"] Oct 31 15:10:16.938774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:16.938731 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" event={"ID":"c71e955c-d595-435b-9702-9ebf6ef2fc70","Type":"ContainerStarted","Data":"48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf"} Oct 31 15:10:16.939223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:16.938792 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" event={"ID":"c71e955c-d595-435b-9702-9ebf6ef2fc70","Type":"ContainerStarted","Data":"8eb331e6c5c93b4163658951eb4f0a900f6b5ee1035f24fb0e6823a707e789a3"} Oct 31 15:10:16.939223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:16.938824 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:16.940422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:16.940397 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" event={"ID":"18a65fdb-497b-4152-938f-854f35e3b818","Type":"ContainerStarted","Data":"25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465"} Oct 31 15:10:16.940569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:16.940450 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:17.063792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:17.063753 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" podUID="1db35cd3-437b-4a02-ae0b-ac45aa570895" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:10:17.063935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:17.063766 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" podUID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:10:17.064953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:17.064910 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" podStartSLOduration=2.06489664 podStartE2EDuration="2.06489664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:17.06089475 +0000 UTC m=+1462.680446161" watchObservedRunningTime="2026-10-31 15:10:17.06489664 +0000 UTC m=+1462.684448041" Oct 31 15:10:17.491778 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:10:17.491741 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:10:19.113894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:19.113853 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:20.808155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:20.808124 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:20.957564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:20.950951 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:21.311241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:21.311208 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:21.470337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:21.470263 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:21.727707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:21.727360 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:24.049694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:24.049642 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" podStartSLOduration=10.049628339 podStartE2EDuration="10.049628339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:17.187706711 +0000 UTC m=+1462.807258136" watchObservedRunningTime="2026-10-31 15:10:24.049628339 +0000 UTC m=+1469.669179750" Oct 31 15:10:24.052353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:24.052327 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx"] Oct 31 15:10:24.103755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:24.103729 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52965934-sw4lx"] Oct 31 15:10:24.497549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:24.497506 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb2c10f-aaa7-4679-bd27-3410940c2202" path="/var/lib/kubelet/pods/3bb2c10f-aaa7-4679-bd27-3410940c2202/volumes" Oct 31 15:10:25.201714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.201682 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt"] Oct 31 15:10:25.207714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.207696 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.262992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.262964 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt"] Oct 31 15:10:25.279115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.279077 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvk9f\" (UniqueName: \"kubernetes.io/projected/b9b696ce-c1b7-47ed-831e-3515a8867261-kube-api-access-wvk9f\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.279280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.279198 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.279280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.279262 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.308685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.308658 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:10:25.308685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.308674 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:10:25.311680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.311657 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:10:25.311680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.311658 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:10:25.380513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.380462 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvk9f\" (UniqueName: \"kubernetes.io/projected/b9b696ce-c1b7-47ed-831e-3515a8867261-kube-api-access-wvk9f\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.380699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.380554 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.380699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.380612 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.381208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.381187 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.383141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.383121 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.438752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.438723 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvk9f\" (UniqueName: \"kubernetes.io/projected/b9b696ce-c1b7-47ed-831e-3515a8867261-kube-api-access-wvk9f\") pod \"maas-api-revoke-sub-c4601702-7kgjt\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.523099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.523073 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:25.902073 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:10:25.902039 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b696ce_c1b7_47ed_831e_3515a8867261.slice/crio-42ac6ba48d8f8b9ec31c58a1808a22ca2612f7e1b1188e61f1ad2b7689771343 WatchSource:0}: Error finding container 42ac6ba48d8f8b9ec31c58a1808a22ca2612f7e1b1188e61f1ad2b7689771343: Status 404 returned error can't find the container with id 42ac6ba48d8f8b9ec31c58a1808a22ca2612f7e1b1188e61f1ad2b7689771343 Oct 31 15:10:25.902591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.902572 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt"] Oct 31 15:10:25.935781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:25.935761 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:26.098500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:26.085755 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp"] Oct 31 15:10:26.098500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:26.085822 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30b9277-8z4sp"] Oct 31 15:10:26.105809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:26.104783 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" event={"ID":"b9b696ce-c1b7-47ed-831e-3515a8867261","Type":"ContainerStarted","Data":"42ac6ba48d8f8b9ec31c58a1808a22ca2612f7e1b1188e61f1ad2b7689771343"} Oct 31 15:10:26.498163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:26.498128 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306ad577-b94e-45ca-98a1-ed3eaf7af4ed" path="/var/lib/kubelet/pods/306ad577-b94e-45ca-98a1-ed3eaf7af4ed/volumes" Oct 31 15:10:26.565272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:26.565188 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:27.062940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:27.062892 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" podUID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:10:27.062940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:27.062935 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" podUID="1db35cd3-437b-4a02-ae0b-ac45aa570895" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:10:27.115116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:27.115031 4865 generic.go:356] "Generic (PLEG): container finished" podID="b9b696ce-c1b7-47ed-831e-3515a8867261" containerID="05d9b8738b2a52ae9ae963aea7d58a2976db6711d8ddbbf08604bcefd91f2d5b" exitCode=0 Oct 31 15:10:27.115284 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:27.115166 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" event={"ID":"b9b696ce-c1b7-47ed-831e-3515a8867261","Type":"ContainerDied","Data":"05d9b8738b2a52ae9ae963aea7d58a2976db6711d8ddbbf08604bcefd91f2d5b"} Oct 31 15:10:27.926745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:27.926715 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9"] Oct 31 15:10:27.963670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:27.963643 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:10:27.963921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:27.963903 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:10:28.000768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.000743 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:28.044678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.044085 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9"] Oct 31 15:10:28.273539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.273509 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:28.321114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.321094 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:28.425457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.423832 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:28.453948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.453803 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b696ce-c1b7-47ed-831e-3515a8867261-kube-api-access-wvk9f\" (UniqueName: \"kubernetes.io/projected/b9b696ce-c1b7-47ed-831e-3515a8867261-kube-api-access-wvk9f\") pod \"b9b696ce-c1b7-47ed-831e-3515a8867261\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " Oct 31 15:10:28.454114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.454074 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-xks-ca\") pod \"b9b696ce-c1b7-47ed-831e-3515a8867261\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " Oct 31 15:10:28.454214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.454196 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-openshift-ca\") pod \"b9b696ce-c1b7-47ed-831e-3515a8867261\" (UID: \"b9b696ce-c1b7-47ed-831e-3515a8867261\") " Oct 31 15:10:28.457108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.457072 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-openshift-ca" pod "b9b696ce-c1b7-47ed-831e-3515a8867261" (UID: "b9b696ce-c1b7-47ed-831e-3515a8867261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:10:28.460395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.460351 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-xks-ca" pod "b9b696ce-c1b7-47ed-831e-3515a8867261" (UID: "b9b696ce-c1b7-47ed-831e-3515a8867261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:10:28.468134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.467658 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b696ce-c1b7-47ed-831e-3515a8867261-kube-api-access-wvk9f" pod "b9b696ce-c1b7-47ed-831e-3515a8867261" (UID: "b9b696ce-c1b7-47ed-831e-3515a8867261"). InnerVolumeSpecName "kube-api-access-wvk9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:10:28.586514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.585215 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:28.586514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.585265 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvk9f\" (UniqueName: \"kubernetes.io/projected/b9b696ce-c1b7-47ed-831e-3515a8867261-kube-api-access-wvk9f\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:28.586514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.585284 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b696ce-c1b7-47ed-831e-3515a8867261-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:28.589072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:28.588755 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:29.145601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.145562 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" Oct 31 15:10:29.145601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.145581 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt" event={"ID":"b9b696ce-c1b7-47ed-831e-3515a8867261","Type":"ContainerDied","Data":"42ac6ba48d8f8b9ec31c58a1808a22ca2612f7e1b1188e61f1ad2b7689771343"} Oct 31 15:10:29.146018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.145668 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ac6ba48d8f8b9ec31c58a1808a22ca2612f7e1b1188e61f1ad2b7689771343" Oct 31 15:10:29.786591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.784977 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:10:29.902927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.902893 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db35cd3-437b-4a02-ae0b-ac45aa570895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db35cd3-437b-4a02-ae0b-ac45aa570895-plugins-config-volume\") pod \"1db35cd3-437b-4a02-ae0b-ac45aa570895\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " Oct 31 15:10:29.903669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.903003 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db35cd3-437b-4a02-ae0b-ac45aa570895-kube-api-access-x4prd\" (UniqueName: \"kubernetes.io/projected/1db35cd3-437b-4a02-ae0b-ac45aa570895-kube-api-access-x4prd\") pod \"1db35cd3-437b-4a02-ae0b-ac45aa570895\" (UID: \"1db35cd3-437b-4a02-ae0b-ac45aa570895\") " Oct 31 15:10:29.904535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.904509 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db35cd3-437b-4a02-ae0b-ac45aa570895-plugins-config-volume" pod "1db35cd3-437b-4a02-ae0b-ac45aa570895" (UID: "1db35cd3-437b-4a02-ae0b-ac45aa570895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:10:29.906822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.906796 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db35cd3-437b-4a02-ae0b-ac45aa570895-kube-api-access-x4prd" pod "1db35cd3-437b-4a02-ae0b-ac45aa570895" (UID: "1db35cd3-437b-4a02-ae0b-ac45aa570895"). InnerVolumeSpecName "kube-api-access-x4prd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:10:29.984194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:29.984169 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:10:30.005042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.005011 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db35cd3-437b-4a02-ae0b-ac45aa570895-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:30.005042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.005039 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4prd\" (UniqueName: \"kubernetes.io/projected/1db35cd3-437b-4a02-ae0b-ac45aa570895-kube-api-access-x4prd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:30.106670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.106108 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8877cb4f-44d1-456e-a403-0ff085dfdcfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8877cb4f-44d1-456e-a403-0ff085dfdcfd-plugins-config-volume\") pod \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " Oct 31 15:10:30.106670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.106162 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8877cb4f-44d1-456e-a403-0ff085dfdcfd-kube-api-access-49qlk\" (UniqueName: \"kubernetes.io/projected/8877cb4f-44d1-456e-a403-0ff085dfdcfd-kube-api-access-49qlk\") pod \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\" (UID: \"8877cb4f-44d1-456e-a403-0ff085dfdcfd\") " Oct 31 15:10:30.106670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.106572 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8877cb4f-44d1-456e-a403-0ff085dfdcfd-plugins-config-volume" pod "8877cb4f-44d1-456e-a403-0ff085dfdcfd" (UID: "8877cb4f-44d1-456e-a403-0ff085dfdcfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:10:30.107044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.106950 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8877cb4f-44d1-456e-a403-0ff085dfdcfd-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:30.108655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.108633 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8877cb4f-44d1-456e-a403-0ff085dfdcfd-kube-api-access-49qlk" pod "8877cb4f-44d1-456e-a403-0ff085dfdcfd" (UID: "8877cb4f-44d1-456e-a403-0ff085dfdcfd"). InnerVolumeSpecName "kube-api-access-49qlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:10:30.160613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.160574 4865 generic.go:356] "Generic (PLEG): container finished" podID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" containerID="a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55" exitCode=0 Oct 31 15:10:30.161047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.160631 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" Oct 31 15:10:30.161047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.160662 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" event={"ID":"8877cb4f-44d1-456e-a403-0ff085dfdcfd","Type":"ContainerDied","Data":"a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55"} Oct 31 15:10:30.161047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.160720 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb" event={"ID":"8877cb4f-44d1-456e-a403-0ff085dfdcfd","Type":"ContainerDied","Data":"71d5d886f62af3b7ea1631af15ffdd12fe1107b410c53251a175f36dd00492bb"} Oct 31 15:10:30.161047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.160750 4865 scope.go:122] "RemoveContainer" containerID="a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55" Oct 31 15:10:30.162403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.162383 4865 generic.go:356] "Generic (PLEG): container finished" podID="1db35cd3-437b-4a02-ae0b-ac45aa570895" containerID="bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5" exitCode=0 Oct 31 15:10:30.162558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.162436 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" Oct 31 15:10:30.162558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.162485 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" event={"ID":"1db35cd3-437b-4a02-ae0b-ac45aa570895","Type":"ContainerDied","Data":"bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5"} Oct 31 15:10:30.162558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.162540 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd" event={"ID":"1db35cd3-437b-4a02-ae0b-ac45aa570895","Type":"ContainerDied","Data":"ac52a0cdee8dc5e29c6f26df99738f4a1727b2106796127f41e9017cb909d96f"} Oct 31 15:10:30.173138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.173118 4865 scope.go:122] "RemoveContainer" containerID="a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55" Oct 31 15:10:30.173417 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:10:30.173395 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55\": container with ID starting with a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55 not found: ID does not exist" containerID="a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55" Oct 31 15:10:30.173538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.173425 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55"} err="failed to get container status \"a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55\": rpc error: code = NotFound desc = could not find container \"a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55\": container with ID starting with a49b2eab49f1bd65edecafff827d0be28f83e12ede9aea61e6675dae16f56c55 not found: ID does not exist" Oct 31 15:10:30.173538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.173442 4865 scope.go:122] "RemoveContainer" containerID="bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5" Oct 31 15:10:30.183551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.183345 4865 scope.go:122] "RemoveContainer" containerID="bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5" Oct 31 15:10:30.183634 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:10:30.183619 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5\": container with ID starting with bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5 not found: ID does not exist" containerID="bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5" Oct 31 15:10:30.183679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.183641 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5"} err="failed to get container status \"bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5\": rpc error: code = NotFound desc = could not find container \"bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5\": container with ID starting with bdc9b2fd8355facb530f99e5351423e0f9490555867ebbe74493f72d13a075f5 not found: ID does not exist" Oct 31 15:10:30.208706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.208684 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qlk\" (UniqueName: \"kubernetes.io/projected/8877cb4f-44d1-456e-a403-0ff085dfdcfd-kube-api-access-49qlk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:30.329893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.329865 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd"] Oct 31 15:10:30.333873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.333839 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-r5mkd"] Oct 31 15:10:30.443839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.443758 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb"] Oct 31 15:10:30.447832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.447808 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-8d5vb"] Oct 31 15:10:30.498310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.498284 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db35cd3-437b-4a02-ae0b-ac45aa570895" path="/var/lib/kubelet/pods/1db35cd3-437b-4a02-ae0b-ac45aa570895/volumes" Oct 31 15:10:30.498741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:30.498728 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" path="/var/lib/kubelet/pods/8877cb4f-44d1-456e-a403-0ff085dfdcfd/volumes" Oct 31 15:10:31.042602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:31.042575 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:31.673538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:31.673509 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:31.920913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:31.905709 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.139749 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6"] Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.140776 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1db35cd3-437b-4a02-ae0b-ac45aa570895" containerName="payload-processing" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.140798 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db35cd3-437b-4a02-ae0b-ac45aa570895" containerName="payload-processing" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.140855 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" containerName="payload-pre-processing" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.140863 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" containerName="payload-pre-processing" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.140883 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b9b696ce-c1b7-47ed-831e-3515a8867261" containerName="revoke-keys" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.140890 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b696ce-c1b7-47ed-831e-3515a8867261" containerName="revoke-keys" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.141168 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8877cb4f-44d1-456e-a403-0ff085dfdcfd" containerName="payload-pre-processing" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.141191 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b696ce-c1b7-47ed-831e-3515a8867261" containerName="revoke-keys" Oct 31 15:10:32.144497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.141211 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db35cd3-437b-4a02-ae0b-ac45aa570895" containerName="payload-processing" Oct 31 15:10:32.181512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.176517 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.261490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.260762 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-644a8a-dockercfg-p67mn\"" Oct 31 15:10:32.280784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.262519 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7cede-ede5-4f4a-94d3-3af7ed7af956-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.280784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.262713 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6qd8\" (UniqueName: \"kubernetes.io/projected/14e7cede-ede5-4f4a-94d3-3af7ed7af956-kube-api-access-v6qd8\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.343532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.337946 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-644a8a\"" Oct 31 15:10:32.375496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.366207 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7cede-ede5-4f4a-94d3-3af7ed7af956-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.375496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.366522 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6qd8\" (UniqueName: \"kubernetes.io/projected/14e7cede-ede5-4f4a-94d3-3af7ed7af956-kube-api-access-v6qd8\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.375496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.370058 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7cede-ede5-4f4a-94d3-3af7ed7af956-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.411941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.380424 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6"] Oct 31 15:10:32.426699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.426665 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx"] Oct 31 15:10:32.477677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.476923 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.486493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.482873 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx"] Oct 31 15:10:32.505076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.504510 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6qd8\" (UniqueName: \"kubernetes.io/projected/14e7cede-ede5-4f4a-94d3-3af7ed7af956-kube-api-access-v6qd8\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.613714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.613677 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.614005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.613986 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffp5\" (UniqueName: \"kubernetes.io/projected/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-kube-api-access-9ffp5\") pod \"payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.715659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.715624 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffp5\" (UniqueName: \"kubernetes.io/projected/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-kube-api-access-9ffp5\") pod \"payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.716062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.715706 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.716401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.716380 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.734209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.734183 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:32.773647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.773618 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffp5\" (UniqueName: \"kubernetes.io/projected/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-kube-api-access-9ffp5\") pod \"payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.837650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.837376 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:32.931364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:32.927494 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" podUID="9a01dcf2-1da5-41d6-a045-7061a9d6c004" containerName="payload-processing" containerID="cri-o://218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520" gracePeriod=25 Oct 31 15:10:33.044921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.044889 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" podUID="74f6f824-4ec9-4c5c-9372-3a316820801f" containerName="payload-pre-processing" containerID="cri-o://4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844" gracePeriod=25 Oct 31 15:10:33.122417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.116099 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6"] Oct 31 15:10:33.214903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.214629 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:10:33.229831 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:10:33.229808 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cb6015_17c8_4476_9a88_4d7a2d693c4c.slice/crio-aa1a54ccf002d0438ab948ddbec21b9c8daf22fccc4287dfb657faa150f9e816 WatchSource:0}: Error finding container aa1a54ccf002d0438ab948ddbec21b9c8daf22fccc4287dfb657faa150f9e816: Status 404 returned error can't find the container with id aa1a54ccf002d0438ab948ddbec21b9c8daf22fccc4287dfb657faa150f9e816 Oct 31 15:10:33.231418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.231393 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx"] Oct 31 15:10:33.240866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.240836 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:33.327672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.327398 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a01dcf2-1da5-41d6-a045-7061a9d6c004-kube-api-access-gq9kj\" (UniqueName: \"kubernetes.io/projected/9a01dcf2-1da5-41d6-a045-7061a9d6c004-kube-api-access-gq9kj\") pod \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " Oct 31 15:10:33.327672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.327451 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a01dcf2-1da5-41d6-a045-7061a9d6c004-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a01dcf2-1da5-41d6-a045-7061a9d6c004-plugins-config-volume\") pod \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\" (UID: \"9a01dcf2-1da5-41d6-a045-7061a9d6c004\") " Oct 31 15:10:33.327986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.327917 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a01dcf2-1da5-41d6-a045-7061a9d6c004-plugins-config-volume" pod "9a01dcf2-1da5-41d6-a045-7061a9d6c004" (UID: "9a01dcf2-1da5-41d6-a045-7061a9d6c004"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:10:33.330023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.328605 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a01dcf2-1da5-41d6-a045-7061a9d6c004-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:33.334498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.334445 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a01dcf2-1da5-41d6-a045-7061a9d6c004-kube-api-access-gq9kj" pod "9a01dcf2-1da5-41d6-a045-7061a9d6c004" (UID: "9a01dcf2-1da5-41d6-a045-7061a9d6c004"). InnerVolumeSpecName "kube-api-access-gq9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:10:33.341522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.341456 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:10:33.381660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.381632 4865 generic.go:356] "Generic (PLEG): container finished" podID="74f6f824-4ec9-4c5c-9372-3a316820801f" containerID="4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844" exitCode=0 Oct 31 15:10:33.381804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.381678 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" Oct 31 15:10:33.381804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.381717 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" event={"ID":"74f6f824-4ec9-4c5c-9372-3a316820801f","Type":"ContainerDied","Data":"4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844"} Oct 31 15:10:33.381804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.381768 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9" event={"ID":"74f6f824-4ec9-4c5c-9372-3a316820801f","Type":"ContainerDied","Data":"5e57d210bf78b10d57730c4135e2fb1f3d84b5235364aa98cb447cb7f9eb0025"} Oct 31 15:10:33.381804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.381797 4865 scope.go:122] "RemoveContainer" containerID="4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844" Oct 31 15:10:33.383612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.383590 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" event={"ID":"14e7cede-ede5-4f4a-94d3-3af7ed7af956","Type":"ContainerStarted","Data":"f1005e15930ec34624cbb2ad7c12d8c78a2abd815d2c9575b515ee393235d4f2"} Oct 31 15:10:33.383738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.383643 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" event={"ID":"14e7cede-ede5-4f4a-94d3-3af7ed7af956","Type":"ContainerStarted","Data":"5a3decf7eedfd9c9c21997fbeabab086e6f8524843c718885a5b3cd50a4de218"} Oct 31 15:10:33.383738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.383669 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:33.385354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.385336 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" event={"ID":"e9cb6015-17c8-4476-9a88-4d7a2d693c4c","Type":"ContainerStarted","Data":"67e5fb58e9d096abc3cb810d695c1cd2df281724616007ef685e9242fda92610"} Oct 31 15:10:33.385493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.385373 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:33.385493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.385390 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" event={"ID":"e9cb6015-17c8-4476-9a88-4d7a2d693c4c","Type":"ContainerStarted","Data":"aa1a54ccf002d0438ab948ddbec21b9c8daf22fccc4287dfb657faa150f9e816"} Oct 31 15:10:33.386855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.386838 4865 generic.go:356] "Generic (PLEG): container finished" podID="9a01dcf2-1da5-41d6-a045-7061a9d6c004" containerID="218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520" exitCode=0 Oct 31 15:10:33.386947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.386869 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" Oct 31 15:10:33.386947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.386919 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" event={"ID":"9a01dcf2-1da5-41d6-a045-7061a9d6c004","Type":"ContainerDied","Data":"218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520"} Oct 31 15:10:33.387064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.386969 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9" event={"ID":"9a01dcf2-1da5-41d6-a045-7061a9d6c004","Type":"ContainerDied","Data":"669e202a3b0332cdf3cdb72e79adef275322f9b5fe32121f0fdd78135d6dad10"} Oct 31 15:10:33.393048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.393029 4865 scope.go:122] "RemoveContainer" containerID="4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844" Oct 31 15:10:33.393339 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:10:33.393322 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844\": container with ID starting with 4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844 not found: ID does not exist" containerID="4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844" Oct 31 15:10:33.393407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.393348 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844"} err="failed to get container status \"4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844\": rpc error: code = NotFound desc = could not find container \"4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844\": container with ID starting with 4b63001f61529ba154a8d1cfa4526f575cb1234bdf611be868bd4120978c6844 not found: ID does not exist" Oct 31 15:10:33.393407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.393365 4865 scope.go:122] "RemoveContainer" containerID="218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520" Oct 31 15:10:33.403500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.403484 4865 scope.go:122] "RemoveContainer" containerID="218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520" Oct 31 15:10:33.403739 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:10:33.403723 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520\": container with ID starting with 218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520 not found: ID does not exist" containerID="218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520" Oct 31 15:10:33.403798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.403748 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520"} err="failed to get container status \"218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520\": rpc error: code = NotFound desc = could not find container \"218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520\": container with ID starting with 218665924af5c271a3619b8482b9112fb3538ec7d1bbfc7e0d7f0758bb88e520 not found: ID does not exist" Oct 31 15:10:33.430523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.430500 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9kj\" (UniqueName: \"kubernetes.io/projected/9a01dcf2-1da5-41d6-a045-7061a9d6c004-kube-api-access-gq9kj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:33.531525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.531495 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f6f824-4ec9-4c5c-9372-3a316820801f-kube-api-access-tkbcv\" (UniqueName: \"kubernetes.io/projected/74f6f824-4ec9-4c5c-9372-3a316820801f-kube-api-access-tkbcv\") pod \"74f6f824-4ec9-4c5c-9372-3a316820801f\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " Oct 31 15:10:33.531754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.531740 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f6f824-4ec9-4c5c-9372-3a316820801f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6f824-4ec9-4c5c-9372-3a316820801f-plugins-config-volume\") pod \"74f6f824-4ec9-4c5c-9372-3a316820801f\" (UID: \"74f6f824-4ec9-4c5c-9372-3a316820801f\") " Oct 31 15:10:33.532034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.532008 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f6f824-4ec9-4c5c-9372-3a316820801f-plugins-config-volume" pod "74f6f824-4ec9-4c5c-9372-3a316820801f" (UID: "74f6f824-4ec9-4c5c-9372-3a316820801f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:10:33.532196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.532181 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f6f824-4ec9-4c5c-9372-3a316820801f-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:33.533734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.533713 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f6f824-4ec9-4c5c-9372-3a316820801f-kube-api-access-tkbcv" pod "74f6f824-4ec9-4c5c-9372-3a316820801f" (UID: "74f6f824-4ec9-4c5c-9372-3a316820801f"). InnerVolumeSpecName "kube-api-access-tkbcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:10:33.572979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.572915 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9"] Oct 31 15:10:33.581209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.581187 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3356b-7864c46bd7-pdws9"] Oct 31 15:10:33.633883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.633858 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkbcv\" (UniqueName: \"kubernetes.io/projected/74f6f824-4ec9-4c5c-9372-3a316820801f-kube-api-access-tkbcv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:10:33.693598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.693544 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" podStartSLOduration=1.6935290200000002 podStartE2EDuration="1.69352902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:33.690312793 +0000 UTC m=+1479.309864203" watchObservedRunningTime="2026-10-31 15:10:33.69352902 +0000 UTC m=+1479.313080528" Oct 31 15:10:33.983719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.979840 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" podStartSLOduration=1.9798212259999999 podStartE2EDuration="1.979821226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:33.860722389 +0000 UTC m=+1479.480273800" watchObservedRunningTime="2026-10-31 15:10:33.979821226 +0000 UTC m=+1479.599372638" Oct 31 15:10:33.995497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:33.991557 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9"] Oct 31 15:10:34.008740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:34.000143 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3356b-7d88f7b57f-gmjh9"] Oct 31 15:10:34.498445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:34.498415 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f6f824-4ec9-4c5c-9372-3a316820801f" path="/var/lib/kubelet/pods/74f6f824-4ec9-4c5c-9372-3a316820801f/volumes" Oct 31 15:10:34.498900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:34.498884 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a01dcf2-1da5-41d6-a045-7061a9d6c004" path="/var/lib/kubelet/pods/9a01dcf2-1da5-41d6-a045-7061a9d6c004/volumes" Oct 31 15:10:35.917508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:35.917395 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:40.569620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:40.569588 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:40.735503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:40.732089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:40.877543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:40.868613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:41.058487 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:41.058435 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk"] Oct 31 15:10:41.104048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:41.104022 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d69aa4d-hvqfk"] Oct 31 15:10:42.513223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:42.513194 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2eaec2-31c9-47e0-93bd-1595ad194a3d" path="/var/lib/kubelet/pods/1a2eaec2-31c9-47e0-93bd-1595ad194a3d/volumes" Oct 31 15:10:44.407976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:44.407949 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:10:44.408413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:44.408289 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:10:45.519958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:45.519901 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:46.050377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:46.050344 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj"] Oct 31 15:10:46.102190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:46.102152 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0db5cee-dzmqj"] Oct 31 15:10:46.497130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:46.497098 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733e18fa-df5f-4218-8540-60bd58be1b02" path="/var/lib/kubelet/pods/733e18fa-df5f-4218-8540-60bd58be1b02/volumes" Oct 31 15:10:47.490565 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:10:47.490530 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:10:48.049288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:48.049135 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp"] Oct 31 15:10:48.101483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:48.101440 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a747ae-hmkzp"] Oct 31 15:10:48.498525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:48.498491 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98489ae7-ed99-4396-93a6-bd58892f8e17" path="/var/lib/kubelet/pods/98489ae7-ed99-4396-93a6-bd58892f8e17/volumes" Oct 31 15:10:49.615711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:49.615658 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:51.412684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:51.412640 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:53.050960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:53.050903 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:54.387343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:54.387308 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:54.704009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:54.703937 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:10:54.709914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:54.709887 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:10:54.712219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:54.712194 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:10:54.717999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:54.717980 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:10:55.222413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.222382 4865 scope.go:122] "RemoveContainer" containerID="56f8e0980388c318841a5e7dec5b28b8db210b9c5d37bb0ed76ce0b598e5bc56" Oct 31 15:10:55.233633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.233609 4865 scope.go:122] "RemoveContainer" containerID="2587ae9f5c62f12eaf058d68e6516dc71c22be1bda23d7ccb2a8f5c08a230441" Oct 31 15:10:55.243832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.243814 4865 scope.go:122] "RemoveContainer" containerID="2bb58c511d13acfd1102168d921d5644f6ac7a6448f95710bfcfdce9eacf5752" Oct 31 15:10:55.252863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.252840 4865 scope.go:122] "RemoveContainer" containerID="5a2c6a74ea0c70e4a17d150fb785fdb5ee81668364825f15d3e337fadfedbc54" Oct 31 15:10:55.262812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.262795 4865 scope.go:122] "RemoveContainer" containerID="9c99a75f2c1b036dec3dca2b8b39a595ecf1f839c7fff52492b9b50e39d0ea20" Oct 31 15:10:55.271667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.271646 4865 scope.go:122] "RemoveContainer" containerID="37d0664dc71b23f4f1dba8074d70a577e00f09d0949be0e5345c97137fcd859a" Oct 31 15:10:55.280407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.280391 4865 scope.go:122] "RemoveContainer" containerID="8f9db0dcf3d1b82854aac6998a89c7b2c67a4d65da0033d95729849d52a63a00" Oct 31 15:10:55.290052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:55.290036 4865 scope.go:122] "RemoveContainer" containerID="19f82563988d15e0e026e575bdf8dcaeb047ad73c84e0d98ef37160fb7a10b04" Oct 31 15:10:56.048861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:56.048797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:56.189498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:56.186164 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:56.330556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:56.328948 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:57.570399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:57.570362 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:10:57.830541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:57.829797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:58.290494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:58.282001 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:10:58.411495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:58.410550 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp"] Oct 31 15:10:58.428494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:58.422752 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" containerID="cri-o://13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203" gracePeriod=30 Oct 31 15:10:59.130494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:10:59.123392 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:01.312495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:01.307918 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:03.833924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.833897 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:11:03.860656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.860623 4865 generic.go:356] "Generic (PLEG): container finished" podID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerID="13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203" exitCode=0 Oct 31 15:11:03.860809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.860668 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" Oct 31 15:11:03.860809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.860699 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" event={"ID":"967ee42d-2dd6-4a02-9ad1-9f67ddae877b","Type":"ContainerDied","Data":"13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203"} Oct 31 15:11:03.860809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.860775 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp" event={"ID":"967ee42d-2dd6-4a02-9ad1-9f67ddae877b","Type":"ContainerDied","Data":"90376822487af31258ed50ae2dc354a0242d2da502b1f3b090c1669993348e48"} Oct 31 15:11:03.860809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.860797 4865 scope.go:122] "RemoveContainer" containerID="13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203" Oct 31 15:11:03.874790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.874002 4865 scope.go:122] "RemoveContainer" containerID="aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303" Oct 31 15:11:03.881234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881214 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-credential-socket\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881268 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-token\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-token\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881307 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-podinfo\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881330 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-data\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881460 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-envoy\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881461 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-credential-socket" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:03.881581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881527 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istiod-ca-cert\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881630 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-data" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:03.881707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881688 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-certs\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881727 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-kube-api-access-wn9k4\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-kube-api-access-wn9k4\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881769 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-socket\") pod \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\" (UID: \"967ee42d-2dd6-4a02-9ad1-9f67ddae877b\") " Oct 31 15:11:03.881960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.881933 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-certs" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:03.882154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.882043 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-socket" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:03.882154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.882062 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istiod-ca-cert" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:03.882552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.882531 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.882651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.882556 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.882651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.882571 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.882651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.882584 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.882651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.882597 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.884328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.884306 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-podinfo" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:03.884415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.884325 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-envoy" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:03.884415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.884379 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-token" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:03.884623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.884600 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-kube-api-access-wn9k4" pod "967ee42d-2dd6-4a02-9ad1-9f67ddae877b" (UID: "967ee42d-2dd6-4a02-9ad1-9f67ddae877b"). InnerVolumeSpecName "kube-api-access-wn9k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:03.885654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.885630 4865 scope.go:122] "RemoveContainer" containerID="13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203" Oct 31 15:11:03.885937 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:03.885920 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203\": container with ID starting with 13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203 not found: ID does not exist" containerID="13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203" Oct 31 15:11:03.886005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.885943 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203"} err="failed to get container status \"13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203\": rpc error: code = NotFound desc = could not find container \"13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203\": container with ID starting with 13446b4ec81c6b2fe5ba184ec40629dafa37a135ef14fb1d7e76830339b22203 not found: ID does not exist" Oct 31 15:11:03.886005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.885961 4865 scope.go:122] "RemoveContainer" containerID="aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303" Oct 31 15:11:03.886223 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:03.886208 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303\": container with ID starting with aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303 not found: ID does not exist" containerID="aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303" Oct 31 15:11:03.886271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.886228 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303"} err="failed to get container status \"aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303\": rpc error: code = NotFound desc = could not find container \"aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303\": container with ID starting with aa588ac0dd2c3d28c09152b01dc438ec2271c5a3b4034016eb54021ff365d303 not found: ID does not exist" Oct 31 15:11:03.983687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.983658 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.983687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.983685 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.983687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.983695 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn9k4\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-kube-api-access-wn9k4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:03.983925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:03.983704 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967ee42d-2dd6-4a02-9ad1-9f67ddae877b-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:04.230010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:04.229980 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh"] Oct 31 15:11:04.288209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:04.288181 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp"] Oct 31 15:11:04.318551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:04.310561 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3356b-openshift-default-784dcddc7-2p2sp"] Oct 31 15:11:04.519493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:04.517808 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" path="/var/lib/kubelet/pods/967ee42d-2dd6-4a02-9ad1-9f67ddae877b/volumes" Oct 31 15:11:04.574685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:04.574647 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:04.946384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:04.946282 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:05.067567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:05.067525 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:06.216678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:06.216642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:07.047215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:07.047171 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs"] Oct 31 15:11:07.098961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:07.098922 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-769ddd41-t9kgs"] Oct 31 15:11:07.485642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:07.485602 4865 patch_prober.go:29] interesting pod/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:11:07.486161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:07.485681 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:08.498730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:08.498697 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d44e54a-4c49-4cb3-9b76-239d210ee158" path="/var/lib/kubelet/pods/5d44e54a-4c49-4cb3-9b76-239d210ee158/volumes" Oct 31 15:11:09.231355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.231301 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" podUID="fa54e817-61df-4128-a183-bf589a13c9b9" containerName="payload-processing" containerID="cri-o://4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0" gracePeriod=25 Oct 31 15:11:09.520485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.520441 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:11:09.657089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.657061 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa54e817-61df-4128-a183-bf589a13c9b9-kube-api-access-8rvx4\" (UniqueName: \"kubernetes.io/projected/fa54e817-61df-4128-a183-bf589a13c9b9-kube-api-access-8rvx4\") pod \"fa54e817-61df-4128-a183-bf589a13c9b9\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " Oct 31 15:11:09.657296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.657172 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa54e817-61df-4128-a183-bf589a13c9b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa54e817-61df-4128-a183-bf589a13c9b9-plugins-config-volume\") pod \"fa54e817-61df-4128-a183-bf589a13c9b9\" (UID: \"fa54e817-61df-4128-a183-bf589a13c9b9\") " Oct 31 15:11:09.657866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.657841 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa54e817-61df-4128-a183-bf589a13c9b9-plugins-config-volume" pod "fa54e817-61df-4128-a183-bf589a13c9b9" (UID: "fa54e817-61df-4128-a183-bf589a13c9b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:09.659900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.659877 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa54e817-61df-4128-a183-bf589a13c9b9-kube-api-access-8rvx4" pod "fa54e817-61df-4128-a183-bf589a13c9b9" (UID: "fa54e817-61df-4128-a183-bf589a13c9b9"). InnerVolumeSpecName "kube-api-access-8rvx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:09.759632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.759554 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa54e817-61df-4128-a183-bf589a13c9b9-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:09.759632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:09.759584 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rvx4\" (UniqueName: \"kubernetes.io/projected/fa54e817-61df-4128-a183-bf589a13c9b9-kube-api-access-8rvx4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:10.062030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.061946 4865 generic.go:356] "Generic (PLEG): container finished" podID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerID="9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba" exitCode=0 Oct 31 15:11:10.062030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.062020 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" event={"ID":"0e639f49-cb86-4ab5-a18e-efe152020a2a","Type":"ContainerDied","Data":"9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba"} Oct 31 15:11:10.062572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.062551 4865 scope.go:122] "RemoveContainer" containerID="9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba" Oct 31 15:11:10.063673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.063651 4865 generic.go:356] "Generic (PLEG): container finished" podID="fa54e817-61df-4128-a183-bf589a13c9b9" containerID="4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0" exitCode=0 Oct 31 15:11:10.063778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.063682 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" event={"ID":"fa54e817-61df-4128-a183-bf589a13c9b9","Type":"ContainerDied","Data":"4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0"} Oct 31 15:11:10.063778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.063709 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" Oct 31 15:11:10.063778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.063736 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh" event={"ID":"fa54e817-61df-4128-a183-bf589a13c9b9","Type":"ContainerDied","Data":"caae2812558b88c4ae00ff997b813827a07305639f019647860b926b1e8adf7f"} Oct 31 15:11:10.063778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.063767 4865 scope.go:122] "RemoveContainer" containerID="4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0" Oct 31 15:11:10.064130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.064099 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:10.064200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.064186 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:10.064252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.064236 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:10.076317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.076297 4865 scope.go:122] "RemoveContainer" containerID="4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0" Oct 31 15:11:10.076651 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:10.076633 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0\": container with ID starting with 4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0 not found: ID does not exist" containerID="4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0" Oct 31 15:11:10.076712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.076660 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0"} err="failed to get container status \"4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0\": rpc error: code = NotFound desc = could not find container \"4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0\": container with ID starting with 4702629c240fb4ce29ab68a30be11d459b57e52c93a7ebccc36b7c5bc157a8a0 not found: ID does not exist" Oct 31 15:11:10.300966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.300774 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh"] Oct 31 15:11:10.307798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.307764 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf5d51-85844cf4f7-bbgkh"] Oct 31 15:11:10.503125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.503083 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa54e817-61df-4128-a183-bf589a13c9b9" path="/var/lib/kubelet/pods/fa54e817-61df-4128-a183-bf589a13c9b9/volumes" Oct 31 15:11:10.521569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:10.521536 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:11.073700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:11.073666 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" event={"ID":"0e639f49-cb86-4ab5-a18e-efe152020a2a","Type":"ContainerStarted","Data":"396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5"} Oct 31 15:11:11.248014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:11.247982 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:11.616494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:11.614796 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:11:11.616494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:11.614932 4865 patch_prober.go:29] interesting pod/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:11:11.616494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:11.614974 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 15:11:12.613495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:12.611852 4865 patch_prober.go:29] interesting pod/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:11:12.613495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:12.611927 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 15:11:13.615495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:13.610825 4865 patch_prober.go:29] interesting pod/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:11:13.615495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:13.610884 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 15:11:14.615493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:14.614829 4865 patch_prober.go:29] interesting pod/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:11:14.615493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:14.614887 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 15:11:15.619067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:15.619024 4865 patch_prober.go:29] interesting pod/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:11:15.619632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:15.619104 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 15:11:16.006023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:16.005874 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:16.139004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:16.138969 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:16.613885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:16.613858 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:11:16.614360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:16.614342 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:11:16.615126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:16.615109 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:11:17.490363 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:17.490328 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:11:20.253095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:20.253061 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:21.784980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:21.784951 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:22.000294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:22.000263 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:22.138921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:22.138822 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:22.318686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:22.316109 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4"] Oct 31 15:11:22.347116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:22.347078 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5695787b-p9rl4"] Oct 31 15:11:22.496794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:22.496765 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bf3bce-410f-4027-a485-e2cd386ff4ef" path="/var/lib/kubelet/pods/64bf3bce-410f-4027-a485-e2cd386ff4ef/volumes" Oct 31 15:11:26.049535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:26.049507 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh"] Oct 31 15:11:26.089278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:26.089242 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:26.101445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:26.101408 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569e34c-fbdjh"] Oct 31 15:11:26.238918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:26.238886 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:26.497576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:26.497545 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa6534b-4e97-4111-8c31-c1bfadee55ac" path="/var/lib/kubelet/pods/2aa6534b-4e97-4111-8c31-c1bfadee55ac/volumes" Oct 31 15:11:27.239355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:27.239322 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:28.047905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:28.047876 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv"] Oct 31 15:11:28.098697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:28.098669 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866444d4-lgvnv"] Oct 31 15:11:28.551235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:28.551201 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf15517-6d78-429d-8dca-d002ecfc356f" path="/var/lib/kubelet/pods/4cf15517-6d78-429d-8dca-d002ecfc356f/volumes" Oct 31 15:11:28.567192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:28.567159 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:11:29.829656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:29.829622 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq"] Oct 31 15:11:29.830158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:29.829825 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" containerID="cri-o://5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25" gracePeriod=30 Oct 31 15:11:29.852697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:29.852663 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:29.881293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:29.881264 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q"] Oct 31 15:11:29.881553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:29.881526 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" containerID="cri-o://e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8" gracePeriod=30 Oct 31 15:11:29.885767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:29.885737 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj"] Oct 31 15:11:29.886021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:29.885995 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" containerID="cri-o://f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed" gracePeriod=30 Oct 31 15:11:30.052907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:30.052878 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:32.134963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.134932 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:32.466893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.466857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:32.673256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.673218 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:32.741194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.741110 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9"] Oct 31 15:11:32.744949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.744917 4865 cpu_manager.go:414] "containerMap: removing container" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" Oct 31 15:11:32.745137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.745123 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" Oct 31 15:11:32.745256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.745244 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9a01dcf2-1da5-41d6-a045-7061a9d6c004" containerName="payload-processing" Oct 31 15:11:32.745337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.745326 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a01dcf2-1da5-41d6-a045-7061a9d6c004" containerName="payload-processing" Oct 31 15:11:32.745553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.745535 4865 cpu_manager.go:414] "containerMap: removing container" podUID="74f6f824-4ec9-4c5c-9372-3a316820801f" containerName="payload-pre-processing" Oct 31 15:11:32.745666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.745654 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f6f824-4ec9-4c5c-9372-3a316820801f" containerName="payload-pre-processing" Oct 31 15:11:32.745845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.745833 4865 cpu_manager.go:414] "containerMap: removing container" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" Oct 31 15:11:32.745942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.745930 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" Oct 31 15:11:32.746257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.746243 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fa54e817-61df-4128-a183-bf589a13c9b9" containerName="payload-processing" Oct 31 15:11:32.746366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.746347 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa54e817-61df-4128-a183-bf589a13c9b9" containerName="payload-processing" Oct 31 15:11:32.747875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.747800 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa54e817-61df-4128-a183-bf589a13c9b9" containerName="payload-processing" Oct 31 15:11:32.748025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.748014 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" Oct 31 15:11:32.748123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.748113 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a01dcf2-1da5-41d6-a045-7061a9d6c004" containerName="payload-processing" Oct 31 15:11:32.748204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.748195 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f6f824-4ec9-4c5c-9372-3a316820801f" containerName="payload-pre-processing" Oct 31 15:11:32.995381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.995308 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9"] Oct 31 15:11:32.995716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:32.995655 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.008268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.008209 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq"] Oct 31 15:11:33.009436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.009415 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="967ee42d-2dd6-4a02-9ad1-9f67ddae877b" containerName="istio-proxy" Oct 31 15:11:33.013128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.013107 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq"] Oct 31 15:11:33.013231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.013189 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.101194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.101169 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5695787b-dockercfg-wz72v\"" Oct 31 15:11:33.101346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.101192 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5695787b\"" Oct 31 15:11:33.105744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.105715 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfs94\" (UniqueName: \"kubernetes.io/projected/84c45d05-3186-46e6-993c-e3d87ac0b3d2-kube-api-access-xfs94\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.105854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.105820 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c45d05-3186-46e6-993c-e3d87ac0b3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.106025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.106011 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825c2c2-c391-4ab5-b8ac-3e06344a342d-plugins-config-volume\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-445hq\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.106112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.106095 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4khnr\" (UniqueName: \"kubernetes.io/projected/1825c2c2-c391-4ab5-b8ac-3e06344a342d-kube-api-access-4khnr\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-445hq\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.207888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.207859 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825c2c2-c391-4ab5-b8ac-3e06344a342d-plugins-config-volume\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-445hq\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.208285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.207905 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4khnr\" (UniqueName: \"kubernetes.io/projected/1825c2c2-c391-4ab5-b8ac-3e06344a342d-kube-api-access-4khnr\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-445hq\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.208285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.207984 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfs94\" (UniqueName: \"kubernetes.io/projected/84c45d05-3186-46e6-993c-e3d87ac0b3d2-kube-api-access-xfs94\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.208285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.208044 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c45d05-3186-46e6-993c-e3d87ac0b3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.208735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.208715 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825c2c2-c391-4ab5-b8ac-3e06344a342d-plugins-config-volume\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-445hq\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.208869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.208854 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c45d05-3186-46e6-993c-e3d87ac0b3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.265604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.265545 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfs94\" (UniqueName: \"kubernetes.io/projected/84c45d05-3186-46e6-993c-e3d87ac0b3d2-kube-api-access-xfs94\") pod \"payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.265709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.265632 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4khnr\" (UniqueName: \"kubernetes.io/projected/1825c2c2-c391-4ab5-b8ac-3e06344a342d-kube-api-access-4khnr\") pod \"payload-processing-e2e-mt-5695787b-5c7448d59c-445hq\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.314562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.314531 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:33.327094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.327073 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:33.578897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.578819 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:11:33.578897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.578883 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:11:33.700242 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:11:33.700212 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c45d05_3186_46e6_993c_e3d87ac0b3d2.slice/crio-f66369ec7be753f129b9101614d072d7217b12d862ed6ecd1395a1a61463c142 WatchSource:0}: Error finding container f66369ec7be753f129b9101614d072d7217b12d862ed6ecd1395a1a61463c142: Status 404 returned error can't find the container with id f66369ec7be753f129b9101614d072d7217b12d862ed6ecd1395a1a61463c142 Oct 31 15:11:33.701560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.701536 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9"] Oct 31 15:11:33.752347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:33.752318 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq"] Oct 31 15:11:33.758460 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:11:33.758434 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1825c2c2_c391_4ab5_b8ac_3e06344a342d.slice/crio-d9d9ea703af91274c32774f1348ab58bd692e992521f740d9e178ce6e47e9d48 WatchSource:0}: Error finding container d9d9ea703af91274c32774f1348ab58bd692e992521f740d9e178ce6e47e9d48: Status 404 returned error can't find the container with id d9d9ea703af91274c32774f1348ab58bd692e992521f740d9e178ce6e47e9d48 Oct 31 15:11:34.526686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.526648 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" event={"ID":"84c45d05-3186-46e6-993c-e3d87ac0b3d2","Type":"ContainerStarted","Data":"ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b"} Oct 31 15:11:34.527174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.526862 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:34.527174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.526895 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" event={"ID":"84c45d05-3186-46e6-993c-e3d87ac0b3d2","Type":"ContainerStarted","Data":"f66369ec7be753f129b9101614d072d7217b12d862ed6ecd1395a1a61463c142"} Oct 31 15:11:34.528386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.528362 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" event={"ID":"1825c2c2-c391-4ab5-b8ac-3e06344a342d","Type":"ContainerStarted","Data":"19946beba32880764e7416fc58de34810409d6a6e2b4726b2bb6048ad09e0448"} Oct 31 15:11:34.528503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.528404 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:34.528503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.528421 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" event={"ID":"1825c2c2-c391-4ab5-b8ac-3e06344a342d","Type":"ContainerStarted","Data":"d9d9ea703af91274c32774f1348ab58bd692e992521f740d9e178ce6e47e9d48"} Oct 31 15:11:34.645632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.645578 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" podStartSLOduration=2.645564438 podStartE2EDuration="2.645564438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:34.640720377 +0000 UTC m=+1540.260271801" watchObservedRunningTime="2026-10-31 15:11:34.645564438 +0000 UTC m=+1540.265115859" Oct 31 15:11:34.762856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:34.762791 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" podStartSLOduration=2.762771118 podStartE2EDuration="2.762771118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:34.755662228 +0000 UTC m=+1540.375213643" watchObservedRunningTime="2026-10-31 15:11:34.762771118 +0000 UTC m=+1540.382322531" Oct 31 15:11:35.203137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.203106 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:11:35.236229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236200 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-credential-socket\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236292 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-envoy\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236402 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-certs\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236487 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-credential-socket" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.236550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236512 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-kube-api-access-89999\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-kube-api-access-89999\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236550 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-socket\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236577 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-podinfo\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236620 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-data\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236691 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-token\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-token\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236775 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef24796-a595-498c-bbbc-7b6fabdccc49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef24796-a595-498c-bbbc-7b6fabdccc49-istiod-ca-cert\") pod \"0ef24796-a595-498c-bbbc-7b6fabdccc49\" (UID: \"0ef24796-a595-498c-bbbc-7b6fabdccc49\") " Oct 31 15:11:35.236916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236839 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-socket" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.236993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236968 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-data" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.237044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.236748 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-certs" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.237307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.237274 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef24796-a595-498c-bbbc-7b6fabdccc49-istiod-ca-cert" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:35.237510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.237494 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.237632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.237515 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.237632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.237531 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.237632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.237543 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.237632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.237556 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef24796-a595-498c-bbbc-7b6fabdccc49-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.239197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.239175 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-token" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:35.239367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.239345 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-envoy" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.239523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.239503 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-podinfo" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:35.239681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.239661 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-kube-api-access-89999" pod "0ef24796-a595-498c-bbbc-7b6fabdccc49" (UID: "0ef24796-a595-498c-bbbc-7b6fabdccc49"). InnerVolumeSpecName "kube-api-access-89999". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:35.259783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.259748 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 15:11:35.271735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.271652 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:11:35.338123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338096 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35246052-d285-42ea-9fbf-5ff3f818d064-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35246052-d285-42ea-9fbf-5ff3f818d064-istio-podinfo\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.338303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338138 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6608d87-3bf4-433e-a0b1-04fc46f49384-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6608d87-3bf4-433e-a0b1-04fc46f49384-istiod-ca-cert\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.338303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338167 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-istio-token\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-istio-token\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.338303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338244 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-data\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.338303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338276 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-certs\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.338611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338589 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-certs" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.338679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338599 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6608d87-3bf4-433e-a0b1-04fc46f49384-istiod-ca-cert" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:35.338679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338586 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-data" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.338846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338821 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-credential-socket\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338919 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-certs\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338955 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-podinfo\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.338987 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-socket\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339014 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-data\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339043 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-credential-socket\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339092 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-socket\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339159 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-envoy\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339192 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-envoy\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339269 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35246052-d285-42ea-9fbf-5ff3f818d064-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35246052-d285-42ea-9fbf-5ff3f818d064-istiod-ca-cert\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339340 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-certs" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339361 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-kube-api-access-2wr6k\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-kube-api-access-2wr6k\") pod \"35246052-d285-42ea-9fbf-5ff3f818d064\" (UID: \"35246052-d285-42ea-9fbf-5ff3f818d064\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339397 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-kube-api-access-qjc6p\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-kube-api-access-qjc6p\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339436 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-token\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-token\") pod \"d6608d87-3bf4-433e-a0b1-04fc46f49384\" (UID: \"d6608d87-3bf4-433e-a0b1-04fc46f49384\") " Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339638 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-socket" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.339780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339685 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-data" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.339867 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-credential-socket" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340062 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-socket" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340345 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35246052-d285-42ea-9fbf-5ff3f818d064-istiod-ca-cert" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340355 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89999\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-kube-api-access-89999\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340378 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340393 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6608d87-3bf4-433e-a0b1-04fc46f49384-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340407 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340421 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340434 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340450 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340463 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340497 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340510 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340524 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.340634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340539 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef24796-a595-498c-bbbc-7b6fabdccc49-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.341360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.340890 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-credential-socket" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.341748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.341726 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-istio-token" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:35.342262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.342226 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35246052-d285-42ea-9fbf-5ff3f818d064-istio-podinfo" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:35.343264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.343237 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-token" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:35.343605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.343580 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-kube-api-access-qjc6p" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "kube-api-access-qjc6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:35.344405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.344369 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-kube-api-access-2wr6k" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "kube-api-access-2wr6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:35.345001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.344939 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-podinfo" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:35.345513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.345438 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-envoy" pod "35246052-d285-42ea-9fbf-5ff3f818d064" (UID: "35246052-d285-42ea-9fbf-5ff3f818d064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.345616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.345569 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-envoy" pod "d6608d87-3bf4-433e-a0b1-04fc46f49384" (UID: "d6608d87-3bf4-433e-a0b1-04fc46f49384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:35.445502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445389 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wr6k\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-kube-api-access-2wr6k\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445419 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjc6p\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-kube-api-access-qjc6p\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445428 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445440 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35246052-d285-42ea-9fbf-5ff3f818d064-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445454 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35246052-d285-42ea-9fbf-5ff3f818d064-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445521 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445540 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445554 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6608d87-3bf4-433e-a0b1-04fc46f49384-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445578 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35246052-d285-42ea-9fbf-5ff3f818d064-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.445882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.445592 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35246052-d285-42ea-9fbf-5ff3f818d064-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:35.538645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.538609 4865 generic.go:356] "Generic (PLEG): container finished" podID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerID="e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8" exitCode=0 Oct 31 15:11:35.539053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.538679 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" Oct 31 15:11:35.539053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.538679 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" event={"ID":"0ef24796-a595-498c-bbbc-7b6fabdccc49","Type":"ContainerDied","Data":"e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8"} Oct 31 15:11:35.539053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.538735 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q" event={"ID":"0ef24796-a595-498c-bbbc-7b6fabdccc49","Type":"ContainerDied","Data":"996002e856f7545fb66d4f161c3e047694589e8c499b3b0902723c45a36d3dfc"} Oct 31 15:11:35.539053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.538756 4865 scope.go:122] "RemoveContainer" containerID="e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8" Oct 31 15:11:35.540752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.540726 4865 generic.go:356] "Generic (PLEG): container finished" podID="35246052-d285-42ea-9fbf-5ff3f818d064" containerID="5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25" exitCode=0 Oct 31 15:11:35.540880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.540773 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" Oct 31 15:11:35.540880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.540809 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" event={"ID":"35246052-d285-42ea-9fbf-5ff3f818d064","Type":"ContainerDied","Data":"5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25"} Oct 31 15:11:35.540880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.540853 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq" event={"ID":"35246052-d285-42ea-9fbf-5ff3f818d064","Type":"ContainerDied","Data":"0cfd5c3c12132da809216abf5df1462615cf4d4fb64d1d82689aac97177cc6ef"} Oct 31 15:11:35.542586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.542563 4865 generic.go:356] "Generic (PLEG): container finished" podID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerID="f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed" exitCode=0 Oct 31 15:11:35.542716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.542636 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" Oct 31 15:11:35.542716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.542652 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" event={"ID":"d6608d87-3bf4-433e-a0b1-04fc46f49384","Type":"ContainerDied","Data":"f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed"} Oct 31 15:11:35.542716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.542701 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj" event={"ID":"d6608d87-3bf4-433e-a0b1-04fc46f49384","Type":"ContainerDied","Data":"0f02fabfe12aa1c77a9a787d4f3d5a878579f75e5cbda01c010a22eea431dba0"} Oct 31 15:11:35.567828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.560054 4865 scope.go:122] "RemoveContainer" containerID="e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8" Oct 31 15:11:35.567828 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:35.560956 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8\": container with ID starting with e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8 not found: ID does not exist" containerID="e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8" Oct 31 15:11:35.567828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.561012 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8"} err="failed to get container status \"e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8\": rpc error: code = NotFound desc = could not find container \"e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8\": container with ID starting with e9c88ea22982b3281a2093271d066425f16b223386ccf387bc3fd04d568b2ff8 not found: ID does not exist" Oct 31 15:11:35.567828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.561038 4865 scope.go:122] "RemoveContainer" containerID="5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25" Oct 31 15:11:35.579195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.579174 4865 scope.go:122] "RemoveContainer" containerID="5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25" Oct 31 15:11:35.580134 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:35.580056 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25\": container with ID starting with 5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25 not found: ID does not exist" containerID="5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25" Oct 31 15:11:35.580134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.580098 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25"} err="failed to get container status \"5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25\": rpc error: code = NotFound desc = could not find container \"5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25\": container with ID starting with 5b305d394edd5d322636e7150b1173ed4571656826816624ec39718b54b97f25 not found: ID does not exist" Oct 31 15:11:35.580134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.580125 4865 scope.go:122] "RemoveContainer" containerID="f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed" Oct 31 15:11:35.593142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.593122 4865 scope.go:122] "RemoveContainer" containerID="f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed" Oct 31 15:11:35.593511 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:35.593485 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed\": container with ID starting with f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed not found: ID does not exist" containerID="f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed" Oct 31 15:11:35.593607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.593516 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed"} err="failed to get container status \"f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed\": rpc error: code = NotFound desc = could not find container \"f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed\": container with ID starting with f88b6a86cd026aa0e468220ad0c68e757acfd01a76e6840c6b2fea01baaa63ed not found: ID does not exist" Oct 31 15:11:35.810188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.810139 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q"] Oct 31 15:11:35.814032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.814006 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mrc8q"] Oct 31 15:11:35.924620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.924594 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq"] Oct 31 15:11:35.929895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:35.929872 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4ctjq"] Oct 31 15:11:36.042241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:36.042211 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj"] Oct 31 15:11:36.046346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:36.046324 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-m2bwj"] Oct 31 15:11:36.497616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:36.497586 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" path="/var/lib/kubelet/pods/0ef24796-a595-498c-bbbc-7b6fabdccc49/volumes" Oct 31 15:11:36.498076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:36.498062 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" path="/var/lib/kubelet/pods/35246052-d285-42ea-9fbf-5ff3f818d064/volumes" Oct 31 15:11:36.498543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:36.498530 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" path="/var/lib/kubelet/pods/d6608d87-3bf4-433e-a0b1-04fc46f49384/volumes" Oct 31 15:11:37.056479 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.056437 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w"] Oct 31 15:11:37.057108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057093 4865 cpu_manager.go:414] "containerMap: removing container" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" Oct 31 15:11:37.057157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057110 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" Oct 31 15:11:37.057157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057128 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" Oct 31 15:11:37.057157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057133 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" Oct 31 15:11:37.057157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057154 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" Oct 31 15:11:37.057279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057160 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" Oct 31 15:11:37.057279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057252 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef24796-a595-498c-bbbc-7b6fabdccc49" containerName="istio-proxy" Oct 31 15:11:37.057279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057266 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="35246052-d285-42ea-9fbf-5ff3f818d064" containerName="istio-proxy" Oct 31 15:11:37.057279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.057275 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6608d87-3bf4-433e-a0b1-04fc46f49384" containerName="istio-proxy" Oct 31 15:11:37.062404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.062385 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.107614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.107579 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq"] Oct 31 15:11:37.111300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.111279 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-617c8ee1-h7mpq"] Oct 31 15:11:37.114273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.114242 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh"] Oct 31 15:11:37.118497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.118461 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.154908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.154835 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-644a8a-openshift-default-dockercfg-5qxv9\"" Oct 31 15:11:37.159574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.159552 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w"] Oct 31 15:11:37.167870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.167846 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfvn\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-kube-api-access-8mfvn\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.167886 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168034 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168073 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168106 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168142 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168168 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168214 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fae054-db19-4628-9430-283e5164ebdb-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168258 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168293 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168319 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168378 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwnx\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-kube-api-access-vfwnx\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168406 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adb8cf5-05ef-49af-ac31-714008b9f57c-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168561 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168599 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168657 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.168703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168699 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.168903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.168734 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fae054-db19-4628-9430-283e5164ebdb-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.178672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.178647 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w"] Oct 31 15:11:37.184114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.184093 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.201813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.201792 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq"] Oct 31 15:11:37.206323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.206305 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.209762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.209740 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh"] Oct 31 15:11:37.237292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.237270 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5f17a1-openshift-default-dockercfg-96fx9\"" Oct 31 15:11:37.259052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.259028 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w"] Oct 31 15:11:37.270215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270188 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfvn\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-kube-api-access-8mfvn\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.270334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270234 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f344a2-7bef-4813-982b-e56c2a07e132-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.270334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270256 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.270462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270350 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.270462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270423 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59658\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-kube-api-access-59658\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.270462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270464 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.270668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270519 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.270668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270548 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf190734-87d5-4a23-87d1-91593a593fe4-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.270668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270589 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.270668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270639 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.270875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270851 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.270937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270884 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.270937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270924 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.271039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270954 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.271039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.270981 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.271039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271009 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.271191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271041 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fae054-db19-4628-9430-283e5164ebdb-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.271191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271066 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.271191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271069 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.271191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271094 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.271191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271103 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.271191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271152 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.271510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271381 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.271510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271217 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.271510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271407 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.271510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271495 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.271726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271536 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.271726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271185 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.271726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271676 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xml8\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-kube-api-access-5xml8\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.271879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271729 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwnx\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-kube-api-access-vfwnx\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.271879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271756 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fae054-db19-4628-9430-283e5164ebdb-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.271879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271766 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adb8cf5-05ef-49af-ac31-714008b9f57c-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.271879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271840 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.272090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271952 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.272090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.271985 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf190734-87d5-4a23-87d1-91593a593fe4-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.272090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272023 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.272090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272061 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.272290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272109 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.272290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272153 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.272290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272201 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fae054-db19-4628-9430-283e5164ebdb-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.272290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272234 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.272290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272253 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.272875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272818 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.272875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.272865 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.274455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.274428 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f344a2-7bef-4813-982b-e56c2a07e132-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.276760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.275317 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.276760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.275914 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.276760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.276013 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adb8cf5-05ef-49af-ac31-714008b9f57c-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.277449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.277422 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fae054-db19-4628-9430-283e5164ebdb-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.277860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.277838 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.323522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.323430 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq"] Oct 31 15:11:37.331385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.330895 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.331385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.331048 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.331385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.331337 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfvn\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-kube-api-access-8mfvn\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.332223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.332198 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwnx\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-kube-api-access-vfwnx\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.375950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.375922 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f344a2-7bef-4813-982b-e56c2a07e132-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.375996 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f344a2-7bef-4813-982b-e56c2a07e132-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376050 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376105 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59658\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-kube-api-access-59658\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376140 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376166 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.376219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376192 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf190734-87d5-4a23-87d1-91593a593fe4-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.376425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376244 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376277 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.376425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376317 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.376425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376353 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376385 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.376706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376462 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376527 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.376706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376553 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xml8\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-kube-api-access-5xml8\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.376706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376564 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.376706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376668 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf190734-87d5-4a23-87d1-91593a593fe4-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.377030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376715 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.377030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376718 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.377030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376845 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.377030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376880 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.377179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.377099 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.377179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.377105 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.377435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.377367 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.377435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.377372 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.377435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.376670 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f344a2-7bef-4813-982b-e56c2a07e132-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.377704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.377653 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.378138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.378111 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf190734-87d5-4a23-87d1-91593a593fe4-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.378542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.378518 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.378785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.378766 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f344a2-7bef-4813-982b-e56c2a07e132-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.378969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.378948 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:37.379436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.379418 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.379737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.379719 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf190734-87d5-4a23-87d1-91593a593fe4-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.431896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.431863 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:37.434503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.434444 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.434889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.434865 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59658\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-kube-api-access-59658\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.435043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.435024 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xml8\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-kube-api-access-5xml8\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.435222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.435201 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.505272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.502712 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:37.525077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.524848 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:37.767683 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:11:37.767650 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fae054_db19_4628_9430_283e5164ebdb.slice/crio-3f68c1e051267a3a1ad8ef9890b397304a03f316d5b7ae5fbbdb8edf4b923231 WatchSource:0}: Error finding container 3f68c1e051267a3a1ad8ef9890b397304a03f316d5b7ae5fbbdb8edf4b923231: Status 404 returned error can't find the container with id 3f68c1e051267a3a1ad8ef9890b397304a03f316d5b7ae5fbbdb8edf4b923231 Oct 31 15:11:37.768183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.768162 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w"] Oct 31 15:11:37.770592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.770557 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.770717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.770635 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.770717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.770682 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.831088 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:11:37.831035 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adb8cf5_05ef_49af_ac31_714008b9f57c.slice/crio-67d48a109449251ad14f2be3c6e18d4efcd33c266390ae37bbe8e94bccf1c811 WatchSource:0}: Error finding container 67d48a109449251ad14f2be3c6e18d4efcd33c266390ae37bbe8e94bccf1c811: Status 404 returned error can't find the container with id 67d48a109449251ad14f2be3c6e18d4efcd33c266390ae37bbe8e94bccf1c811 Oct 31 15:11:37.832130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.832109 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh"] Oct 31 15:11:37.833814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.833776 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.833898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.833840 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.833898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.833871 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.835812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.835793 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:37.947638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.947141 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w"] Oct 31 15:11:37.954509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.954459 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq"] Oct 31 15:11:37.989369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.985263 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.989369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.985362 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:37.989369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.985408 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:38.000731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.995405 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:38.000731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.995557 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:38.000731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:37.995608 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:38.508424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.508386 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a1b93f-bfff-4ef0-8af5-ffa45849d077" path="/var/lib/kubelet/pods/a4a1b93f-bfff-4ef0-8af5-ffa45849d077/volumes" Oct 31 15:11:38.579403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.579370 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" event={"ID":"76f344a2-7bef-4813-982b-e56c2a07e132","Type":"ContainerStarted","Data":"141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37"} Oct 31 15:11:38.579618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.579423 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" event={"ID":"76f344a2-7bef-4813-982b-e56c2a07e132","Type":"ContainerStarted","Data":"e7f4226996cf76edb90e54f99aa553518e6d1c74a73e8e7ce6484a46254ec3d9"} Oct 31 15:11:38.581163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.581138 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" event={"ID":"2adb8cf5-05ef-49af-ac31-714008b9f57c","Type":"ContainerStarted","Data":"628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3"} Oct 31 15:11:38.581265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.581181 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" event={"ID":"2adb8cf5-05ef-49af-ac31-714008b9f57c","Type":"ContainerStarted","Data":"67d48a109449251ad14f2be3c6e18d4efcd33c266390ae37bbe8e94bccf1c811"} Oct 31 15:11:38.582895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.582862 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" event={"ID":"24fae054-db19-4628-9430-283e5164ebdb","Type":"ContainerStarted","Data":"c3e7e14e871833775063950fc0fec8eb178c676d72b08e96cbdbf11e8fb3d14b"} Oct 31 15:11:38.583025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.582918 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" event={"ID":"24fae054-db19-4628-9430-283e5164ebdb","Type":"ContainerStarted","Data":"3f68c1e051267a3a1ad8ef9890b397304a03f316d5b7ae5fbbdb8edf4b923231"} Oct 31 15:11:38.584525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.584497 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" event={"ID":"cf190734-87d5-4a23-87d1-91593a593fe4","Type":"ContainerStarted","Data":"35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e"} Oct 31 15:11:38.584612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.584543 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" event={"ID":"cf190734-87d5-4a23-87d1-91593a593fe4","Type":"ContainerStarted","Data":"5fcd0df4db08e10dd6c55798de26c5a7c78f8f870933d1a81533a3ba676ea537"} Oct 31 15:11:38.748495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:38.748310 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podStartSLOduration=1.7482878290000001 podStartE2EDuration="1.748287829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:38.696411932 +0000 UTC m=+1544.315963349" watchObservedRunningTime="2026-10-31 15:11:38.748287829 +0000 UTC m=+1544.367839241" Oct 31 15:11:39.008494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.008184 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podStartSLOduration=2.00815874 podStartE2EDuration="2.00815874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:38.851575617 +0000 UTC m=+1544.471127030" watchObservedRunningTime="2026-10-31 15:11:39.00815874 +0000 UTC m=+1544.627710152" Oct 31 15:11:39.138496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.138204 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podStartSLOduration=2.138189471 podStartE2EDuration="2.138189471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:38.990771978 +0000 UTC m=+1544.610323460" watchObservedRunningTime="2026-10-31 15:11:39.138189471 +0000 UTC m=+1544.757740880" Oct 31 15:11:39.398552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.397736 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:39.402492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.401585 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:39.402492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.401659 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:39.434495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.434222 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:39.434495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.434383 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:39.434495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.434425 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:39.514495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.513502 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:39.515030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.515002 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:39.515117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.515054 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:39.538495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.537786 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:39.547495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.542023 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:39.547495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.542080 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:39.781493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.770654 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:39.831816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.825766 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:11:39.853496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.844424 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:39.875496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.855548 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:39.875496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.855958 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:39.998496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:39.994579 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:11:40.084542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.071571 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:40.118497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.099569 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:40.118497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.100168 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:11:40.118497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.109537 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:11:40.410543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.395218 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:40.410543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.395277 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:40.430988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.429781 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:40.562510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.529182 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:40.562510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.529245 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:40.562510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.541651 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:40.587492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.567245 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:40.587492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.567318 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:40.587492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.567753 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:40.587492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.567796 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:40.710604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.664576 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:11:40.774921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.774856 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:40.846698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.846642 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:11:40.855747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:40.855698 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:11:41.004553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.004201 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:11:41.004553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.004270 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:11:41.004553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.004309 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:11:41.085660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.032708 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:41.085660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.060328 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:11:41.089604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.089557 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:41.090728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.090706 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:11:41.107492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.099972 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:11:41.135389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.123577 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:11:41.135389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.110589 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:41.135740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.135715 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:11:41.136517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.110623 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:11:41.137778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.137759 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:11:41.214495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.205415 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podStartSLOduration=4.205397086 podStartE2EDuration="4.205397086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:39.141238513 +0000 UTC m=+1544.760789925" watchObservedRunningTime="2026-10-31 15:11:41.205397086 +0000 UTC m=+1546.824948501" Oct 31 15:11:41.401555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.398683 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:41.401555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.398747 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:41.552537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.541495 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:41.552537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.541559 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:41.552537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.542252 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:41.552537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.542285 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:41.552537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.544029 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:41.552537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.544061 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:41.685559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.683562 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:41.719507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.710572 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:11:41.809899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.783577 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:11:41.925504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.904578 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:11:41.925504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.920759 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:41.932624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:41.932581 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:11:42.085533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.060560 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:42.116566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.116520 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:11:42.125688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.125627 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:11:42.141603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.141548 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:42.149546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.144346 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:42.218524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.206188 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:42.435726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.435638 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:42.440494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.436530 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:42.451597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.436424 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:42.451861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.451832 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:42.552351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.552312 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:42.552547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.552378 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:42.615450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.615412 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:42.724411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.724373 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:42.752970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.699137 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:11:42.794679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.794632 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:42.912493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.907738 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:42.984551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.923587 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:42.984551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.936069 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t"] Oct 31 15:11:42.984551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.981642 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:11:42.984551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.981660 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:11:42.984551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:42.982785 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" containerID="cri-o://899fee99a5919fa67ea86df5ecea2ce810039b7711a8297bff327de5a56caf5e" gracePeriod=30 Oct 31 15:11:43.015557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.006738 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k"] Oct 31 15:11:43.056782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.056714 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" containerID="cri-o://41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64" gracePeriod=30 Oct 31 15:11:43.112496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.091606 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:11:43.129666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.129623 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:43.130214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.130181 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:11:43.158497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.150714 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:43.158497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.151134 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:43.158497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.152778 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m"] Oct 31 15:11:43.158497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.153001 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" containerID="cri-o://35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916" gracePeriod=30 Oct 31 15:11:43.284553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.251639 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7"] Oct 31 15:11:43.284553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.255719 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" containerID="cri-o://b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2" gracePeriod=30 Oct 31 15:11:43.326513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.318210 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:11:43.433637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.433599 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:43.457581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.457540 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:43.457816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.437565 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:43.457954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.457933 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:43.518495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.513804 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:43.518495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.513861 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:43.518495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.485624 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:43.614549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.573763 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:43.614549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.573825 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:43.659263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.658562 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 15:11:43.756641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.756589 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:11:43.814508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.811578 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:43.986495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.981218 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:11:43.995812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:43.995756 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.116495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.115582 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.131504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.130463 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.131504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.130866 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.160495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.157592 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.162851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.162815 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.329570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.320616 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.382495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.379516 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:44.382495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.379577 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:44.389686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.389646 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:11:44.438420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.438383 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:44.438718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.438690 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:44.522494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.521821 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:44.522494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.521880 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:44.541651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.541616 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:44.542371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.542344 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:44.646045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.645963 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:44.806586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.799614 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.806586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.800382 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:44.865501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.849388 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:45.009366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:44.996801 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:11:45.043686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.036942 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:45.153501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.151092 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:45.153501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.151363 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:11:45.153501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.152679 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:11:45.176610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.176561 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:11:45.190595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.190548 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:45.305082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.304990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:45.350495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.326973 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:45.351158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.351128 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:11:45.379855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.379820 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:45.380175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.380134 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:45.461128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.461095 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:45.462116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.462086 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:45.513083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.513051 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:45.517500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.513313 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:45.562618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.562534 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:45.562872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.562843 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:45.801497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.770648 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:11:45.823720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.822232 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:11:45.826306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.826049 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:45.869504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.867580 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:11:45.941548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.941509 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:11:45.941739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:45.941572 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:11:46.066494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.000551 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:11:46.187720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.125148 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:11:46.187994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.170606 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:11:46.250057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.250011 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:11:46.268016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.267972 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:11:46.402286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.402150 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:11:46.481499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.480334 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:46.492499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.482127 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:11:46.492499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.482787 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:11:46.492499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.482841 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 15:11:46.492499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.484828 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:46.529280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.499654 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:11:46.529280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.527720 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:11:46.529280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.527792 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 31 15:11:46.530806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.530779 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 31 15:11:46.531087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.531056 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 31 15:11:46.543157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.543130 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 15:11:46.543344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.543322 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 15:11:46.601529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.601490 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:46.919201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.901871 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": context deadline exceeded" Oct 31 15:11:46.924935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:46.924891 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:11:47.108285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.108241 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:11:47.108615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.108587 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:11:47.176216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.170791 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:47.232411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.213159 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:11:47.232411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.213330 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:11:47.232411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.213360 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl"] Oct 31 15:11:47.250122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.248884 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl"] Oct 31 15:11:47.250122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.248966 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:11:47.250122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.249010 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:11:47.250122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.249042 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:11:47.250122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.249072 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:11:47.250122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.249100 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:11:47.252287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.252271 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.306891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.306863 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f7251e-openshift-default-dockercfg-9lgvl\"" Oct 31 15:11:47.385600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.385565 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:47.386212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.386010 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:47.387063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.387030 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:11:47.388058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388036 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00341e7-e710-4156-8c80-6947b6d581e7-istio-podinfo\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.388213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-credential-socket\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.388368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388341 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-certs\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.388564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388388 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-data\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.388564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388523 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rn7\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-kube-api-access-p9rn7\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.388713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388678 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-istio-token\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.388856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388837 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-socket\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.388961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.388897 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-envoy\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.389838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.389772 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00341e7-e710-4156-8c80-6947b6d581e7-istiod-ca-cert\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.437250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.437183 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:47.437521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.437502 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:47.438315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.438296 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:11:47.491448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.491421 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-istio-token\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.491771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.491747 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-socket\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.491922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.491905 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-envoy\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.492080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492056 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00341e7-e710-4156-8c80-6947b6d581e7-istiod-ca-cert\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.492202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492130 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-socket\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.492202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492148 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00341e7-e710-4156-8c80-6947b6d581e7-istio-podinfo\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.492337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492226 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-credential-socket\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.492401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492360 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-certs\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.492502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492397 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-data\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.492502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492433 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rn7\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-kube-api-access-p9rn7\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.493014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492810 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00341e7-e710-4156-8c80-6947b6d581e7-istiod-ca-cert\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.493014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492832 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-credential-socket\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.493014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.492958 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-data\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.493316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.493297 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-certs\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.495449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.495421 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00341e7-e710-4156-8c80-6947b6d581e7-istio-podinfo\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.495449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.495447 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-envoy\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.497399 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:47.497374 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:11:47.508531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.508507 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:47.509045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.509026 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:47.509771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.509755 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:11:47.529682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.529664 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:47.529906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.529889 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:47.530498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.530462 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:11:47.549402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.549375 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-istio-token\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.550103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.550086 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rn7\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-kube-api-access-p9rn7\") pod \"e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:47.576896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:47.575031 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:48.016954 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:11:48.016924 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00341e7_e710_4156_8c80_6947b6d581e7.slice/crio-808cfdebf230c23fae3e9af896305a0185872932b6d2e69c86c875c1ac58f191 WatchSource:0}: Error finding container 808cfdebf230c23fae3e9af896305a0185872932b6d2e69c86c875c1ac58f191: Status 404 returned error can't find the container with id 808cfdebf230c23fae3e9af896305a0185872932b6d2e69c86c875c1ac58f191 Oct 31 15:11:48.018075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.018051 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl"] Oct 31 15:11:48.019756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.019725 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:48.019821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.019791 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:48.019855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.019821 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:11:48.146760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.146724 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:11:48.146913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.146791 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:11:48.202094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.202049 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" event={"ID":"a00341e7-e710-4156-8c80-6947b6d581e7","Type":"ContainerStarted","Data":"bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a"} Oct 31 15:11:48.202293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.202142 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" event={"ID":"a00341e7-e710-4156-8c80-6947b6d581e7","Type":"ContainerStarted","Data":"808cfdebf230c23fae3e9af896305a0185872932b6d2e69c86c875c1ac58f191"} Oct 31 15:11:48.205398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.205363 4865 generic.go:356] "Generic (PLEG): container finished" podID="4af3f96a-5606-46b4-950c-8353031d73fe" containerID="899fee99a5919fa67ea86df5ecea2ce810039b7711a8297bff327de5a56caf5e" exitCode=0 Oct 31 15:11:48.206513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.206490 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" event={"ID":"4af3f96a-5606-46b4-950c-8353031d73fe","Type":"ContainerDied","Data":"899fee99a5919fa67ea86df5ecea2ce810039b7711a8297bff327de5a56caf5e"} Oct 31 15:11:48.435837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.435690 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:48.649282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.582838 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:48.711779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.709523 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:11:48.738184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.727087 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:11:48.738184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.727886 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podStartSLOduration=2.7278677609999997 podStartE2EDuration="2.727867761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:48.691070743 +0000 UTC m=+1554.310622156" watchObservedRunningTime="2026-10-31 15:11:48.727867761 +0000 UTC m=+1554.347419174" Oct 31 15:11:48.788141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788110 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istiod-ca-cert\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788171 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-certs\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.788285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788201 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-istio-token\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-istio-token\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.788285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788254 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-credential-socket\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.788483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788289 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-data\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788336 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-certs\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788363 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-socket\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788410 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-kube-api-access-ctjjd\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-kube-api-access-ctjjd\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788459 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-kube-api-access-lmdr9\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-kube-api-access-lmdr9\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.788772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788488 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-certs" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.788772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788531 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istiod-ca-cert" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:48.788772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788575 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-podinfo\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788605 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-credential-socket\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788679 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-envoy\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.788772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788752 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-data" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.788772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788755 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-socket\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788804 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-socket" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788830 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4af3f96a-5606-46b4-950c-8353031d73fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3f96a-5606-46b4-950c-8353031d73fe-istio-podinfo\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788886 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-envoy\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788972 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-token\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-token\") pod \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\" (UID: \"d713974c-1634-4cc0-89c5-cd063ab2a8f8\") " Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.788989 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-certs" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789029 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af3f96a-5606-46b4-950c-8353031d73fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3f96a-5606-46b4-950c-8353031d73fe-istiod-ca-cert\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789038 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-socket" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789084 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-data\") pod \"4af3f96a-5606-46b4-950c-8353031d73fe\" (UID: \"4af3f96a-5606-46b4-950c-8353031d73fe\") " Oct 31 15:11:48.789275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789241 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-credential-socket" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.789864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789508 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-data" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789917 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789944 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789959 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789973 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.789987 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.790000 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.790014 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.790062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.790028 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.791265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.791004 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af3f96a-5606-46b4-950c-8353031d73fe-istiod-ca-cert" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:48.791902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.791627 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-credential-socket" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.793536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.793257 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-envoy" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.887810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.813709 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-podinfo" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:48.892494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.892097 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.892494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.892134 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.892494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.892150 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af3f96a-5606-46b4-950c-8353031d73fe-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.892494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.892166 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af3f96a-5606-46b4-950c-8353031d73fe-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.926267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.922394 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-kube-api-access-ctjjd" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "kube-api-access-ctjjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:48.926267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.922772 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-istio-token" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:48.926267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.922929 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-token" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:48.928296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.927860 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-kube-api-access-lmdr9" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "kube-api-access-lmdr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:48.930365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.929720 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4af3f96a-5606-46b4-950c-8353031d73fe-istio-podinfo" pod "4af3f96a-5606-46b4-950c-8353031d73fe" (UID: "4af3f96a-5606-46b4-950c-8353031d73fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:48.931790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.931741 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-envoy" pod "d713974c-1634-4cc0-89c5-cd063ab2a8f8" (UID: "d713974c-1634-4cc0-89c5-cd063ab2a8f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:48.942263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.942195 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996344 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-data\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996418 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-socket\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996533 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-credential-socket\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996606 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-kube-api-access-hv4p4\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-kube-api-access-hv4p4\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996724 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-envoy\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996777 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec12d57-7084-41cc-8d01-aafab86143d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec12d57-7084-41cc-8d01-aafab86143d5-istiod-ca-cert\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996855 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-istio-token\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-istio-token\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.996978 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-certs\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.997011 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec12d57-7084-41cc-8d01-aafab86143d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec12d57-7084-41cc-8d01-aafab86143d5-istio-podinfo\") pod \"cec12d57-7084-41cc-8d01-aafab86143d5\" (UID: \"cec12d57-7084-41cc-8d01-aafab86143d5\") " Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.997609 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.997629 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctjjd\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-kube-api-access-ctjjd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.997646 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdr9\" (UniqueName: \"kubernetes.io/projected/4af3f96a-5606-46b4-950c-8353031d73fe-kube-api-access-lmdr9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.997660 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.997674 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af3f96a-5606-46b4-950c-8353031d73fe-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:48.999513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:48.997688 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713974c-1634-4cc0-89c5-cd063ab2a8f8-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.000903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.000872 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-data" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.001065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.001048 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-socket" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.001219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.001200 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-credential-socket" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.003950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.003916 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec12d57-7084-41cc-8d01-aafab86143d5-istio-podinfo" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:49.003950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.003920 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-kube-api-access-hv4p4" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "kube-api-access-hv4p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:49.004856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.004821 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-certs" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.005111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.005083 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec12d57-7084-41cc-8d01-aafab86143d5-istiod-ca-cert" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:49.009050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.009021 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-istio-token" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:49.009153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.009094 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-envoy" pod "cec12d57-7084-41cc-8d01-aafab86143d5" (UID: "cec12d57-7084-41cc-8d01-aafab86143d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.029590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.029568 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 15:11:49.098999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.098967 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-socket\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099023 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bde907c-ced4-4f51-9048-8cb571722f91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde907c-ced4-4f51-9048-8cb571722f91-istio-podinfo\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099102 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-credential-socket\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099171 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-certs\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099207 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-kube-api-access-fctkw\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-kube-api-access-fctkw\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099249 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-socket" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.099432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099330 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-envoy\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099398 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bde907c-ced4-4f51-9048-8cb571722f91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde907c-ced4-4f51-9048-8cb571722f91-istiod-ca-cert\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099436 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-credential-socket" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.099570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099497 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-istio-token\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-istio-token\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099615 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-data\") pod \"5bde907c-ced4-4f51-9048-8cb571722f91\" (UID: \"5bde907c-ced4-4f51-9048-8cb571722f91\") " Oct 31 15:11:49.099832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099782 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-certs" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.099955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.099934 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-data" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.100089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100068 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bde907c-ced4-4f51-9048-8cb571722f91-istiod-ca-cert" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100312 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100330 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec12d57-7084-41cc-8d01-aafab86143d5-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100339 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100348 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100357 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100365 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bde907c-ced4-4f51-9048-8cb571722f91-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100373 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100381 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4p4\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-kube-api-access-hv4p4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100390 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100399 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec12d57-7084-41cc-8d01-aafab86143d5-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100406 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec12d57-7084-41cc-8d01-aafab86143d5-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.100364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100417 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.101076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100425 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec12d57-7084-41cc-8d01-aafab86143d5-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.101076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.100433 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.102283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.102256 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-istio-token" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:49.102408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.102292 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bde907c-ced4-4f51-9048-8cb571722f91-istio-podinfo" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:11:49.103560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.103539 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-kube-api-access-fctkw" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "kube-api-access-fctkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:49.103693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.103644 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-envoy" pod "5bde907c-ced4-4f51-9048-8cb571722f91" (UID: "5bde907c-ced4-4f51-9048-8cb571722f91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:11:49.202789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.201951 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bde907c-ced4-4f51-9048-8cb571722f91-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.202789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.202063 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctkw\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-kube-api-access-fctkw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.202789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.202079 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bde907c-ced4-4f51-9048-8cb571722f91-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.202789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.202092 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bde907c-ced4-4f51-9048-8cb571722f91-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:49.218580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.218544 4865 generic.go:356] "Generic (PLEG): container finished" podID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerID="41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64" exitCode=0 Oct 31 15:11:49.218815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.218595 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" event={"ID":"d713974c-1634-4cc0-89c5-cd063ab2a8f8","Type":"ContainerDied","Data":"41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64"} Oct 31 15:11:49.218815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.218652 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" Oct 31 15:11:49.218815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.218664 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k" event={"ID":"d713974c-1634-4cc0-89c5-cd063ab2a8f8","Type":"ContainerDied","Data":"efa2d99d6f508e8ce97bb1dfce333024feb60e632b81e62564617a08b4332374"} Oct 31 15:11:49.218815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.218716 4865 scope.go:122] "RemoveContainer" containerID="41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64" Oct 31 15:11:49.221598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.221567 4865 generic.go:356] "Generic (PLEG): container finished" podID="5bde907c-ced4-4f51-9048-8cb571722f91" containerID="35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916" exitCode=0 Oct 31 15:11:49.221804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.221651 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" Oct 31 15:11:49.221993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.221964 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" event={"ID":"5bde907c-ced4-4f51-9048-8cb571722f91","Type":"ContainerDied","Data":"35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916"} Oct 31 15:11:49.222212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.222195 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m" event={"ID":"5bde907c-ced4-4f51-9048-8cb571722f91","Type":"ContainerDied","Data":"677b81127513c0bd881a70280305776c85632891b994e2f707145e2e6ff153d7"} Oct 31 15:11:49.228904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.228881 4865 generic.go:356] "Generic (PLEG): container finished" podID="cec12d57-7084-41cc-8d01-aafab86143d5" containerID="b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2" exitCode=0 Oct 31 15:11:49.229198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.229157 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" event={"ID":"cec12d57-7084-41cc-8d01-aafab86143d5","Type":"ContainerDied","Data":"b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2"} Oct 31 15:11:49.229298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.229209 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" event={"ID":"cec12d57-7084-41cc-8d01-aafab86143d5","Type":"ContainerDied","Data":"4433ec042e16221921758c5a4576a8d5116cf372bfd3c65dd241842097783c49"} Oct 31 15:11:49.229298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.229289 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7" Oct 31 15:11:49.231495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.231479 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" Oct 31 15:11:49.231636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.231617 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t" event={"ID":"4af3f96a-5606-46b4-950c-8353031d73fe","Type":"ContainerDied","Data":"fc805487165b05db14ee1947ed8df643ec31dd6244efe50438a4d68d8dafbbce"} Oct 31 15:11:49.235167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.235148 4865 scope.go:122] "RemoveContainer" containerID="41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64" Oct 31 15:11:49.235523 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:49.235456 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64\": container with ID starting with 41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64 not found: ID does not exist" containerID="41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64" Oct 31 15:11:49.235600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.235536 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64"} err="failed to get container status \"41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64\": rpc error: code = NotFound desc = could not find container \"41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64\": container with ID starting with 41b6c380184a96999e01993a9761ebfe1d4de3075f7478afd6f3cb4043881a64 not found: ID does not exist" Oct 31 15:11:49.235600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.235560 4865 scope.go:122] "RemoveContainer" containerID="35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916" Oct 31 15:11:49.249633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.249612 4865 scope.go:122] "RemoveContainer" containerID="dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54" Oct 31 15:11:49.259504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.259448 4865 scope.go:122] "RemoveContainer" containerID="35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916" Oct 31 15:11:49.260739 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:49.260709 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916\": container with ID starting with 35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916 not found: ID does not exist" containerID="35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916" Oct 31 15:11:49.260850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.260751 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916"} err="failed to get container status \"35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916\": rpc error: code = NotFound desc = could not find container \"35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916\": container with ID starting with 35e96c9f8bb8518c7a24ebcc751ec8b398e405df17d6f8a2d4c3f7d0c217e916 not found: ID does not exist" Oct 31 15:11:49.260850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.260777 4865 scope.go:122] "RemoveContainer" containerID="dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54" Oct 31 15:11:49.261095 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:49.261066 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54\": container with ID starting with dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54 not found: ID does not exist" containerID="dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54" Oct 31 15:11:49.261157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.261105 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54"} err="failed to get container status \"dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54\": rpc error: code = NotFound desc = could not find container \"dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54\": container with ID starting with dff362b1d0cfe1bc05ff160a3ba1b39a439b9e0b2ed9d544145d564208946e54 not found: ID does not exist" Oct 31 15:11:49.261157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.261128 4865 scope.go:122] "RemoveContainer" containerID="b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2" Oct 31 15:11:49.271670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.271650 4865 scope.go:122] "RemoveContainer" containerID="b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2" Oct 31 15:11:49.271999 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:11:49.271976 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2\": container with ID starting with b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2 not found: ID does not exist" containerID="b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2" Oct 31 15:11:49.272074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.272011 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2"} err="failed to get container status \"b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2\": rpc error: code = NotFound desc = could not find container \"b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2\": container with ID starting with b1a438467af5578e74094f68e08fd96876fecc58367418808773cc1f36d5bdc2 not found: ID does not exist" Oct 31 15:11:49.272074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.272034 4865 scope.go:122] "RemoveContainer" containerID="899fee99a5919fa67ea86df5ecea2ce810039b7711a8297bff327de5a56caf5e" Oct 31 15:11:49.388705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.388661 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t"] Oct 31 15:11:49.393995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.393969 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hg28t"] Oct 31 15:11:49.508557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.508521 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m"] Oct 31 15:11:49.512201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.512179 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zmx7m"] Oct 31 15:11:49.581632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.581604 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:49.622924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.622888 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7"] Oct 31 15:11:49.627902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.627876 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-hbrp7"] Oct 31 15:11:49.739285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.739251 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k"] Oct 31 15:11:49.742957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:49.742932 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9gh4k"] Oct 31 15:11:50.244678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:50.244648 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:50.245642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:50.245618 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:11:50.501535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:50.501415 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" path="/var/lib/kubelet/pods/4af3f96a-5606-46b4-950c-8353031d73fe/volumes" Oct 31 15:11:50.502024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:50.502006 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" path="/var/lib/kubelet/pods/5bde907c-ced4-4f51-9048-8cb571722f91/volumes" Oct 31 15:11:50.502542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:50.502527 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" path="/var/lib/kubelet/pods/cec12d57-7084-41cc-8d01-aafab86143d5/volumes" Oct 31 15:11:50.503338 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:50.503322 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" path="/var/lib/kubelet/pods/d713974c-1634-4cc0-89c5-cd063ab2a8f8/volumes" Oct 31 15:11:51.055009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:51.054969 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l"] Oct 31 15:11:51.058705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:51.058678 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e30cb11a-d5p6l"] Oct 31 15:11:51.956499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:51.956444 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:52.178535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:52.178504 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:52.434808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:52.434775 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:52.501375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:52.501346 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f1c39b-e088-496e-91bf-7abd6bab96e9" path="/var/lib/kubelet/pods/53f1c39b-e088-496e-91bf-7abd6bab96e9/volumes" Oct 31 15:11:53.394695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:53.394667 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:53.530522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:53.530483 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:54.797364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:54.797330 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:55.363378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.363346 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:55.390005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.389971 4865 scope.go:122] "RemoveContainer" containerID="7ff248991d68c2bc2f51e4428dcdd8c3ea6b15921a95e53ebfd8cb1e252b3ef3" Oct 31 15:11:55.400230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.400199 4865 scope.go:122] "RemoveContainer" containerID="93d74ae81e97d67fe2f3e25d98aa8809f738d182dae603e68eb149257779a762" Oct 31 15:11:55.410013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.409995 4865 scope.go:122] "RemoveContainer" containerID="a1ce8b8188bb169e9e373d4bb499c1e63e9b4065a481ff5fc9b82fc0634f80be" Oct 31 15:11:55.420394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.420372 4865 scope.go:122] "RemoveContainer" containerID="62af61d4ea03d2a7aea361552c48e92e30666434109f25083e9deaa2313a5efd" Oct 31 15:11:55.429571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.429552 4865 scope.go:122] "RemoveContainer" containerID="948245c2c71fee8a82f3f9e483a62a36051187787c63a93367bb9da4081ffd4b" Oct 31 15:11:55.438511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.438492 4865 scope.go:122] "RemoveContainer" containerID="7d2718651e568fc59a5ab94cbc5e2f564ac1f59f0b7503b861632797cdacbd26" Oct 31 15:11:55.447939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.447912 4865 scope.go:122] "RemoveContainer" containerID="6aac90021c84e75b78f0d047be6a2e4aac6f0b2572bb706022627e4fb721f035" Oct 31 15:11:55.456775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.456760 4865 scope.go:122] "RemoveContainer" containerID="38818d7fc8e5523f5ff090514401c15ef48e556c144883d8b9114486532d1576" Oct 31 15:11:55.714495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.710920 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.789085 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl"] Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790091 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790109 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790132 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790141 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790192 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790200 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790217 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790224 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790242 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" Oct 31 15:11:55.790535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.790250 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" Oct 31 15:11:55.811867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.797628 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" Oct 31 15:11:55.811867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.797664 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d713974c-1634-4cc0-89c5-cd063ab2a8f8" containerName="istio-proxy" Oct 31 15:11:55.811867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.797692 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af3f96a-5606-46b4-950c-8353031d73fe" containerName="istio-proxy" Oct 31 15:11:55.811867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.797705 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bde907c-ced4-4f51-9048-8cb571722f91" containerName="istio-proxy" Oct 31 15:11:55.811867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.797716 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec12d57-7084-41cc-8d01-aafab86143d5" containerName="istio-proxy" Oct 31 15:11:55.820041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.816777 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:55.896399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.896211 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f7251e\"" Oct 31 15:11:55.896399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.896364 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f7251e-dockercfg-92r5z\"" Oct 31 15:11:55.995548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.985208 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcp4\" (UniqueName: \"kubernetes.io/projected/a1c5d656-4842-47be-bb60-d973c4a9487d-kube-api-access-2gcp4\") pod \"payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:55.995548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:55.985372 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c5d656-4842-47be-bb60-d973c4a9487d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:56.058501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.048001 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl"] Oct 31 15:11:56.108690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.106949 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcp4\" (UniqueName: \"kubernetes.io/projected/a1c5d656-4842-47be-bb60-d973c4a9487d-kube-api-access-2gcp4\") pod \"payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:56.109513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.109463 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c5d656-4842-47be-bb60-d973c4a9487d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:56.110890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.110583 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c5d656-4842-47be-bb60-d973c4a9487d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:56.115503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.113356 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl"] Oct 31 15:11:56.149196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.149166 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.183387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.182884 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcp4\" (UniqueName: \"kubernetes.io/projected/a1c5d656-4842-47be-bb60-d973c4a9487d-kube-api-access-2gcp4\") pod \"payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:56.213460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.209172 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl"] Oct 31 15:11:56.213460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.211939 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c300b0e-e261-468a-9b8b-26bb624b8b28-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7251e-7db596df7-58mfl\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.213460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.212139 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sltq\" (UniqueName: \"kubernetes.io/projected/5c300b0e-e261-468a-9b8b-26bb624b8b28-kube-api-access-2sltq\") pod \"payload-processing-e2e-api-b-f7251e-7db596df7-58mfl\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.314054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.314026 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sltq\" (UniqueName: \"kubernetes.io/projected/5c300b0e-e261-468a-9b8b-26bb624b8b28-kube-api-access-2sltq\") pod \"payload-processing-e2e-api-b-f7251e-7db596df7-58mfl\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.314196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.314182 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c300b0e-e261-468a-9b8b-26bb624b8b28-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7251e-7db596df7-58mfl\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.314899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.314880 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c300b0e-e261-468a-9b8b-26bb624b8b28-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7251e-7db596df7-58mfl\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.372425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.372398 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sltq\" (UniqueName: \"kubernetes.io/projected/5c300b0e-e261-468a-9b8b-26bb624b8b28-kube-api-access-2sltq\") pod \"payload-processing-e2e-api-b-f7251e-7db596df7-58mfl\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.454365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.454334 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:56.525660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.525623 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:56.834153 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:11:56.834125 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c5d656_4842_47be_bb60_d973c4a9487d.slice/crio-ad5089f39dda42c9dec3c5878a82d1fd76a7cad6e0bdf7be21bba789576cc2be WatchSource:0}: Error finding container ad5089f39dda42c9dec3c5878a82d1fd76a7cad6e0bdf7be21bba789576cc2be: Status 404 returned error can't find the container with id ad5089f39dda42c9dec3c5878a82d1fd76a7cad6e0bdf7be21bba789576cc2be Oct 31 15:11:56.834846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.834825 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl"] Oct 31 15:11:56.943766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:56.943740 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl"] Oct 31 15:11:56.944013 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:11:56.943987 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c300b0e_e261_468a_9b8b_26bb624b8b28.slice/crio-0ebdedfb73efda1290a68f1a52d9fad602dac6357dc78c3f1288067e1430559f WatchSource:0}: Error finding container 0ebdedfb73efda1290a68f1a52d9fad602dac6357dc78c3f1288067e1430559f: Status 404 returned error can't find the container with id 0ebdedfb73efda1290a68f1a52d9fad602dac6357dc78c3f1288067e1430559f Oct 31 15:11:57.300919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.300875 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" event={"ID":"a1c5d656-4842-47be-bb60-d973c4a9487d","Type":"ContainerStarted","Data":"1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37"} Oct 31 15:11:57.301112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.300996 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" event={"ID":"a1c5d656-4842-47be-bb60-d973c4a9487d","Type":"ContainerStarted","Data":"ad5089f39dda42c9dec3c5878a82d1fd76a7cad6e0bdf7be21bba789576cc2be"} Oct 31 15:11:57.301163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.301136 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:11:57.302551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.302530 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" event={"ID":"5c300b0e-e261-468a-9b8b-26bb624b8b28","Type":"ContainerStarted","Data":"2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c"} Oct 31 15:11:57.302644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.302565 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" event={"ID":"5c300b0e-e261-468a-9b8b-26bb624b8b28","Type":"ContainerStarted","Data":"0ebdedfb73efda1290a68f1a52d9fad602dac6357dc78c3f1288067e1430559f"} Oct 31 15:11:57.302644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.302584 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:11:57.416146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.416087 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" podStartSLOduration=2.416069301 podStartE2EDuration="2.416069301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:57.416007102 +0000 UTC m=+1563.035558525" watchObservedRunningTime="2026-10-31 15:11:57.416069301 +0000 UTC m=+1563.035620714" Oct 31 15:11:57.570550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.568154 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" podStartSLOduration=2.568131171 podStartE2EDuration="2.568131171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:57.54789056 +0000 UTC m=+1563.167441983" watchObservedRunningTime="2026-10-31 15:11:57.568131171 +0000 UTC m=+1563.187682583" Oct 31 15:11:57.849552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:57.807663 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:11:59.735579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:11:59.735532 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:01.447745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:01.447703 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:01.586984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:01.586956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:02.963422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:02.963389 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:03.453593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:03.453520 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:03.590589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:03.588396 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:04.378334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:04.378288 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:04.850389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:04.850348 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:04.999286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:04.999254 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5"] Oct 31 15:12:05.004865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.004841 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.063733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.062829 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2196e4-openshift-default-dockercfg-jd8zv\"" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.064720 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-socket\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.064810 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-envoy\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.064878 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e480e53c-7fff-4aad-92f7-c52755b17294-istio-podinfo\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.064940 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-certs\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.064996 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-istio-token\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.065045 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-credential-socket\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.065088 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e480e53c-7fff-4aad-92f7-c52755b17294-istiod-ca-cert\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.065115 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzw7x\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-kube-api-access-pzw7x\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.068526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.065170 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-data\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.091835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.090451 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.171601 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-certs\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.171674 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-istio-token\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.171725 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-credential-socket\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.171764 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e480e53c-7fff-4aad-92f7-c52755b17294-istiod-ca-cert\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.171791 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzw7x\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-kube-api-access-pzw7x\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.171841 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-data\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.172032 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-socket\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.172104 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-envoy\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.175571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.172170 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e480e53c-7fff-4aad-92f7-c52755b17294-istio-podinfo\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.186949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.178023 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e480e53c-7fff-4aad-92f7-c52755b17294-istiod-ca-cert\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.186949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.178310 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-certs\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.186949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.184755 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5"] Oct 31 15:12:05.201774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.201536 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-credential-socket\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.211543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.209019 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-data\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.320594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.313375 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-socket\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.320594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.315528 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-istio-token\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.320594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.316272 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzw7x\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-kube-api-access-pzw7x\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.320594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.316932 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e480e53c-7fff-4aad-92f7-c52755b17294-istio-podinfo\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.320594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.317140 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-envoy\") pod \"e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.320594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.319254 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:05.507213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.506878 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:05.831224 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:12:05.831197 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode480e53c_7fff_4aad_92f7_c52755b17294.slice/crio-2e9e452031b165d54c28efc5a93ec1db43dbce089187f9337267e680391e46da WatchSource:0}: Error finding container 2e9e452031b165d54c28efc5a93ec1db43dbce089187f9337267e680391e46da: Status 404 returned error can't find the container with id 2e9e452031b165d54c28efc5a93ec1db43dbce089187f9337267e680391e46da Oct 31 15:12:05.832174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.832145 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5"] Oct 31 15:12:05.833980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.833947 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:05.834070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.834013 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:05.834070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.834043 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:05.915643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:05.915607 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:06.424204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:06.424167 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" event={"ID":"e480e53c-7fff-4aad-92f7-c52755b17294","Type":"ContainerStarted","Data":"1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec"} Oct 31 15:12:06.424401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:06.424218 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" event={"ID":"e480e53c-7fff-4aad-92f7-c52755b17294","Type":"ContainerStarted","Data":"2e9e452031b165d54c28efc5a93ec1db43dbce089187f9337267e680391e46da"} Oct 31 15:12:06.544415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:06.544365 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podStartSLOduration=2.544350569 podStartE2EDuration="2.544350569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:06.543056548 +0000 UTC m=+1572.162607958" watchObservedRunningTime="2026-10-31 15:12:06.544350569 +0000 UTC m=+1572.163901979" Oct 31 15:12:07.320370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:07.320337 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:07.326109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:07.326088 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:07.431223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:07.431196 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:07.432190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:07.432167 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:12:08.352689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:08.352661 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:12:08.353274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:08.353256 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:12:09.362554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:09.362521 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:09.522697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:09.522661 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:09.672134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:09.672053 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:10.207904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:10.207875 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:11.797319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:11.797285 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:13.010414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:13.010383 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:12:13.403601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:13.403523 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:13.554704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:13.533094 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:15.540510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:15.540447 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:15.711676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:15.711633 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:15.845257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:15.845163 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:17.492587 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:12:17.492552 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:12:17.613500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:17.613444 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:18.200878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:18.200846 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:19.030832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.030795 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:19.574979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.574946 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8"] Oct 31 15:12:19.579524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.579502 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.640605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.635853 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-94a889-openshift-default-dockercfg-zcgtq\"" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.665613 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-socket\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.665690 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-envoy\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.665720 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-certs\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.665828 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-podinfo\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.665865 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-token\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.665918 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7md\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-kube-api-access-5r7md\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.665984 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-data\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.666022 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12211dd-e275-441c-8e43-088e4c2bc3af-istiod-ca-cert\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.680496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.666055 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-credential-socket\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785195 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-podinfo\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785251 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-token\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785303 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7md\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-kube-api-access-5r7md\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785358 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-data\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785402 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12211dd-e275-441c-8e43-088e4c2bc3af-istiod-ca-cert\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785437 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-credential-socket\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785576 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-socket\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785641 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-envoy\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.785675 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-certs\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.786981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.786136 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-certs\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.793382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.791645 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12211dd-e275-441c-8e43-088e4c2bc3af-istiod-ca-cert\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.793382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.792371 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-data\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.793382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.792644 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-socket\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.793382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.792911 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-credential-socket\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.817491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.804309 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8"] Oct 31 15:12:19.817491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.807084 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:19.817491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.815946 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-podinfo\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.859004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.858452 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7md\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-kube-api-access-5r7md\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.879671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.866053 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-envoy\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:19.995707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:19.994013 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:20.046395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:20.046354 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-token\") pod \"e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:20.226371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:20.226330 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:20.657955 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:12:20.657928 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12211dd_e275_441c_8e43_088e4c2bc3af.slice/crio-df9bb29e6ab239eb1bb9b34f55900768ce5ec334b35d19077b49aae58099d745 WatchSource:0}: Error finding container df9bb29e6ab239eb1bb9b34f55900768ce5ec334b35d19077b49aae58099d745: Status 404 returned error can't find the container with id df9bb29e6ab239eb1bb9b34f55900768ce5ec334b35d19077b49aae58099d745 Oct 31 15:12:20.658242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:20.658221 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8"] Oct 31 15:12:20.660630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:20.660594 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:20.660747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:20.660681 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:20.660747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:20.660730 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:20.946383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:20.946290 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:21.356485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.356435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:21.612169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.612065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:21.632618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.627714 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" event={"ID":"f12211dd-e275-441c-8e43-088e4c2bc3af","Type":"ContainerStarted","Data":"f47a32101578a89e75362edbedcc883cca0aeb339c3f4a64e69d564b36859ed3"} Oct 31 15:12:21.632618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.627785 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" event={"ID":"f12211dd-e275-441c-8e43-088e4c2bc3af","Type":"ContainerStarted","Data":"df9bb29e6ab239eb1bb9b34f55900768ce5ec334b35d19077b49aae58099d745"} Oct 31 15:12:21.779417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.779375 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f"] Oct 31 15:12:21.783797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.783781 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:21.816780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.816754 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0093-7fa6-440e-afba-524657809b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:21.817022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.816999 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crz4m\" (UniqueName: \"kubernetes.io/projected/926c0093-7fa6-440e-afba-524657809b04-kube-api-access-crz4m\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:21.830123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.830083 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podStartSLOduration=2.83007094 podStartE2EDuration="2.83007094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:21.800203506 +0000 UTC m=+1587.419754928" watchObservedRunningTime="2026-10-31 15:12:21.83007094 +0000 UTC m=+1587.449622350" Oct 31 15:12:21.851244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.851213 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2196e4\"" Oct 31 15:12:21.851384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.851324 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2196e4-dockercfg-kvxwq\"" Oct 31 15:12:21.855273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.855250 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f"] Oct 31 15:12:21.920113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.920036 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0093-7fa6-440e-afba-524657809b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:21.920565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.920537 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crz4m\" (UniqueName: \"kubernetes.io/projected/926c0093-7fa6-440e-afba-524657809b04-kube-api-access-crz4m\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:21.921381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.921349 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0093-7fa6-440e-afba-524657809b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:21.987504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:21.980163 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt"] Oct 31 15:12:22.030712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.030678 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt"] Oct 31 15:12:22.031172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.031039 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.041689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.039790 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crz4m\" (UniqueName: \"kubernetes.io/projected/926c0093-7fa6-440e-afba-524657809b04-kube-api-access-crz4m\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:22.099626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.099448 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:22.231798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.227960 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:22.234414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.234352 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.250925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.250897 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlx9s\" (UniqueName: \"kubernetes.io/projected/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-kube-api-access-hlx9s\") pod \"payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.252190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.251720 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:22.352343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.352308 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.352716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.352686 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlx9s\" (UniqueName: \"kubernetes.io/projected/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-kube-api-access-hlx9s\") pod \"payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.353089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.353063 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.410893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.410854 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlx9s\" (UniqueName: \"kubernetes.io/projected/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-kube-api-access-hlx9s\") pod \"payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.485319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.485228 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx"] Oct 31 15:12:22.548521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.548457 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56"] Oct 31 15:12:22.554634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.554610 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.612549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.612505 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6"] Oct 31 15:12:22.623554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.623510 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56"] Oct 31 15:12:22.647297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.647270 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:22.649570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.649551 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:12:22.654166 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:12:22.654143 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926c0093_7fa6_440e_afba_524657809b04.slice/crio-6cfea3549b3c9825d7d90f12d9dbe39e89152db5728c875e8699a1e4c31c9121 WatchSource:0}: Error finding container 6cfea3549b3c9825d7d90f12d9dbe39e89152db5728c875e8699a1e4c31c9121: Status 404 returned error can't find the container with id 6cfea3549b3c9825d7d90f12d9dbe39e89152db5728c875e8699a1e4c31c9121 Oct 31 15:12:22.655682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.655660 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f"] Oct 31 15:12:22.656045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.656028 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.656111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.656068 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c0901-ef24-4b51-9244-829417ec99cb-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.656235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.656186 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.656772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.656748 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9mp\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-kube-api-access-gv9mp\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.656866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.656805 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.657111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.657090 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.657270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.657249 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.657319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.657295 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.657402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.657385 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.686199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.686154 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:22.759304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759272 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.759803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759337 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c0901-ef24-4b51-9244-829417ec99cb-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.759803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759492 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.759803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759539 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9mp\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-kube-api-access-gv9mp\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.759803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759594 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.759803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759682 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.760070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759884 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.760070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759932 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.760070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.759989 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.760742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.760715 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.760832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.760781 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.761021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.760997 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.761585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.761461 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c0901-ef24-4b51-9244-829417ec99cb-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.761680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.761589 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.764264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.764219 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.764772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.764754 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.818670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.818636 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.822759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.822730 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9mp\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-kube-api-access-gv9mp\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.830373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.830340 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:22.869726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.869696 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:22.982764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:22.977300 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:23.237325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.237286 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt"] Oct 31 15:12:23.399362 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:12:23.399330 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1c0901_ef24_4b51_9244_829417ec99cb.slice/crio-20f26413b490eae30a2f4224849dbc134f1742aa715587062555b9d205a0a5f4 WatchSource:0}: Error finding container 20f26413b490eae30a2f4224849dbc134f1742aa715587062555b9d205a0a5f4: Status 404 returned error can't find the container with id 20f26413b490eae30a2f4224849dbc134f1742aa715587062555b9d205a0a5f4 Oct 31 15:12:23.399738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.399721 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56"] Oct 31 15:12:23.402007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.401974 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:23.402085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.402040 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:23.402085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.402069 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:12:23.656015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.655922 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" event={"ID":"9e1c0901-ef24-4b51-9244-829417ec99cb","Type":"ContainerStarted","Data":"48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e"} Oct 31 15:12:23.656015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.655985 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" event={"ID":"9e1c0901-ef24-4b51-9244-829417ec99cb","Type":"ContainerStarted","Data":"20f26413b490eae30a2f4224849dbc134f1742aa715587062555b9d205a0a5f4"} Oct 31 15:12:23.657654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.657630 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" event={"ID":"926c0093-7fa6-440e-afba-524657809b04","Type":"ContainerStarted","Data":"941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4"} Oct 31 15:12:23.657783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.657679 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" event={"ID":"926c0093-7fa6-440e-afba-524657809b04","Type":"ContainerStarted","Data":"6cfea3549b3c9825d7d90f12d9dbe39e89152db5728c875e8699a1e4c31c9121"} Oct 31 15:12:23.657783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.657732 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:23.659351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.659318 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" event={"ID":"e0b53a6e-dcc2-468d-9b71-afab5b65dccf","Type":"ContainerStarted","Data":"da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02"} Oct 31 15:12:23.659462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.659367 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:23.659462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.659390 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" event={"ID":"e0b53a6e-dcc2-468d-9b71-afab5b65dccf","Type":"ContainerStarted","Data":"9dc5635529505059e34a2830c2033f4b6e2f769bcd4bc99aae910e37d4618845"} Oct 31 15:12:23.775024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.774967 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podStartSLOduration=1.774950484 podStartE2EDuration="1.774950484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:23.773429931 +0000 UTC m=+1589.392981348" watchObservedRunningTime="2026-10-31 15:12:23.774950484 +0000 UTC m=+1589.394501902" Oct 31 15:12:23.870746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.870710 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:23.894403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:23.894347 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" podStartSLOduration=2.894334245 podStartE2EDuration="2.894334245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:23.889752391 +0000 UTC m=+1589.509303802" watchObservedRunningTime="2026-10-31 15:12:23.894334245 +0000 UTC m=+1589.513885655" Oct 31 15:12:24.029491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:24.026919 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" podStartSLOduration=3.026896106 podStartE2EDuration="3.026896106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:24.009898896 +0000 UTC m=+1589.629450318" watchObservedRunningTime="2026-10-31 15:12:24.026896106 +0000 UTC m=+1589.646447518" Oct 31 15:12:24.748228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:24.742541 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:24.872035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:24.871993 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:12:24.872218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:24.872063 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 15:12:24.887925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:24.887888 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:25.869969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:25.869933 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:12:25.870401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:25.869981 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 15:12:26.362492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:26.359914 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:26.870789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:26.870752 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:12:26.871187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:26.870807 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 15:12:27.485832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.485791 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" podUID="e9cb6015-17c8-4476-9a88-4d7a2d693c4c" containerName="payload-processing" containerID="cri-o://67e5fb58e9d096abc3cb810d695c1cd2df281724616007ef685e9242fda92610" gracePeriod=25 Oct 31 15:12:27.616578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.615531 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" podUID="14e7cede-ede5-4f4a-94d3-3af7ed7af956" containerName="payload-pre-processing" containerID="cri-o://f1005e15930ec34624cbb2ad7c12d8c78a2abd815d2c9575b515ee393235d4f2" gracePeriod=25 Oct 31 15:12:27.781346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.781205 4865 generic.go:356] "Generic (PLEG): container finished" podID="14e7cede-ede5-4f4a-94d3-3af7ed7af956" containerID="f1005e15930ec34624cbb2ad7c12d8c78a2abd815d2c9575b515ee393235d4f2" exitCode=0 Oct 31 15:12:27.781346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.781280 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" event={"ID":"14e7cede-ede5-4f4a-94d3-3af7ed7af956","Type":"ContainerDied","Data":"f1005e15930ec34624cbb2ad7c12d8c78a2abd815d2c9575b515ee393235d4f2"} Oct 31 15:12:27.783458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.783436 4865 generic.go:356] "Generic (PLEG): container finished" podID="e9cb6015-17c8-4476-9a88-4d7a2d693c4c" containerID="67e5fb58e9d096abc3cb810d695c1cd2df281724616007ef685e9242fda92610" exitCode=0 Oct 31 15:12:27.783706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.783517 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" event={"ID":"e9cb6015-17c8-4476-9a88-4d7a2d693c4c","Type":"ContainerDied","Data":"67e5fb58e9d096abc3cb810d695c1cd2df281724616007ef685e9242fda92610"} Oct 31 15:12:27.844739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.844713 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:12:27.870960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.870923 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:12:27.872045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.870988 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 15:12:27.962912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.962882 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-kube-api-access-9ffp5\" (UniqueName: \"kubernetes.io/projected/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-kube-api-access-9ffp5\") pod \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " Oct 31 15:12:27.963067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.962999 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-plugins-config-volume\") pod \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\" (UID: \"e9cb6015-17c8-4476-9a88-4d7a2d693c4c\") " Oct 31 15:12:27.963633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.963608 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-plugins-config-volume" pod "e9cb6015-17c8-4476-9a88-4d7a2d693c4c" (UID: "e9cb6015-17c8-4476-9a88-4d7a2d693c4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:27.963843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.963822 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:27.965941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.965919 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-kube-api-access-9ffp5" pod "e9cb6015-17c8-4476-9a88-4d7a2d693c4c" (UID: "e9cb6015-17c8-4476-9a88-4d7a2d693c4c"). InnerVolumeSpecName "kube-api-access-9ffp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:27.983564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:27.983537 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:12:28.065163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.065065 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e7cede-ede5-4f4a-94d3-3af7ed7af956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7cede-ede5-4f4a-94d3-3af7ed7af956-plugins-config-volume\") pod \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " Oct 31 15:12:28.065356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.065270 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e7cede-ede5-4f4a-94d3-3af7ed7af956-kube-api-access-v6qd8\" (UniqueName: \"kubernetes.io/projected/14e7cede-ede5-4f4a-94d3-3af7ed7af956-kube-api-access-v6qd8\") pod \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\" (UID: \"14e7cede-ede5-4f4a-94d3-3af7ed7af956\") " Oct 31 15:12:28.065720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.065693 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e7cede-ede5-4f4a-94d3-3af7ed7af956-plugins-config-volume" pod "14e7cede-ede5-4f4a-94d3-3af7ed7af956" (UID: "14e7cede-ede5-4f4a-94d3-3af7ed7af956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:28.065970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.065952 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7cede-ede5-4f4a-94d3-3af7ed7af956-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:28.066020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.065973 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ffp5\" (UniqueName: \"kubernetes.io/projected/e9cb6015-17c8-4476-9a88-4d7a2d693c4c-kube-api-access-9ffp5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:28.067670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.067653 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e7cede-ede5-4f4a-94d3-3af7ed7af956-kube-api-access-v6qd8" pod "14e7cede-ede5-4f4a-94d3-3af7ed7af956" (UID: "14e7cede-ede5-4f4a-94d3-3af7ed7af956"). InnerVolumeSpecName "kube-api-access-v6qd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:28.168356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.168325 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6qd8\" (UniqueName: \"kubernetes.io/projected/14e7cede-ede5-4f4a-94d3-3af7ed7af956-kube-api-access-v6qd8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:28.793294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.793215 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" event={"ID":"14e7cede-ede5-4f4a-94d3-3af7ed7af956","Type":"ContainerDied","Data":"5a3decf7eedfd9c9c21997fbeabab086e6f8524843c718885a5b3cd50a4de218"} Oct 31 15:12:28.793294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.793276 4865 scope.go:122] "RemoveContainer" containerID="f1005e15930ec34624cbb2ad7c12d8c78a2abd815d2c9575b515ee393235d4f2" Oct 31 15:12:28.793502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.793392 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6" Oct 31 15:12:28.819906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.817751 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" event={"ID":"e9cb6015-17c8-4476-9a88-4d7a2d693c4c","Type":"ContainerDied","Data":"aa1a54ccf002d0438ab948ddbec21b9c8daf22fccc4287dfb657faa150f9e816"} Oct 31 15:12:28.819906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.817858 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx" Oct 31 15:12:28.832691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.832556 4865 scope.go:122] "RemoveContainer" containerID="67e5fb58e9d096abc3cb810d695c1cd2df281724616007ef685e9242fda92610" Oct 31 15:12:28.870027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.870001 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:12:28.870157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.870049 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 15:12:28.982136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.982101 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6"] Oct 31 15:12:28.987516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:28.987493 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-bb4fc64f9-v9sd6"] Oct 31 15:12:29.135445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:29.135347 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx"] Oct 31 15:12:29.141856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:29.141816 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-c547467bf-g6wtx"] Oct 31 15:12:29.930393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:29.930359 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:29.931300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:29.931252 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:29.932051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:29.932012 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:12:30.498723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:30.498691 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e7cede-ede5-4f4a-94d3-3af7ed7af956" path="/var/lib/kubelet/pods/14e7cede-ede5-4f4a-94d3-3af7ed7af956/volumes" Oct 31 15:12:30.499297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:30.499276 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cb6015-17c8-4476-9a88-4d7a2d693c4c" path="/var/lib/kubelet/pods/e9cb6015-17c8-4476-9a88-4d7a2d693c4c/volumes" Oct 31 15:12:31.023758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:31.023727 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:31.223490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:31.221440 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:31.409296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:31.409198 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:33.651405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:33.651371 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg"] Oct 31 15:12:33.651944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:33.651669 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" containerID="cri-o://b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3" gracePeriod=30 Oct 31 15:12:34.760915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:34.760887 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:12:34.761343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:34.760967 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:12:36.236771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:36.236737 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:38.946089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.946064 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:12:38.960902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.960865 4865 generic.go:356] "Generic (PLEG): container finished" podID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerID="b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3" exitCode=0 Oct 31 15:12:38.961044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.960922 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" Oct 31 15:12:38.961186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.961159 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" event={"ID":"c10e6edf-82fb-4819-aa04-21f0db05769b","Type":"ContainerDied","Data":"b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3"} Oct 31 15:12:38.961252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.961235 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg" event={"ID":"c10e6edf-82fb-4819-aa04-21f0db05769b","Type":"ContainerDied","Data":"67e75b669639ddd50b37a3b46152b390310b80729dd5e1fc3a1f8e24376720c9"} Oct 31 15:12:38.961298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.961274 4865 scope.go:122] "RemoveContainer" containerID="b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3" Oct 31 15:12:38.972646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.972624 4865 scope.go:122] "RemoveContainer" containerID="b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3" Oct 31 15:12:38.972955 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:12:38.972921 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3\": container with ID starting with b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3 not found: ID does not exist" containerID="b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3" Oct 31 15:12:38.973081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.972965 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3"} err="failed to get container status \"b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3\": rpc error: code = NotFound desc = could not find container \"b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3\": container with ID starting with b964efe0523b533b725cad27f5b04daccf30c888ba442fdfe4fe81e56669e3f3 not found: ID does not exist" Oct 31 15:12:38.998934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.998902 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-token\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-token\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.998949 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-certs\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.998980 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-kube-api-access-nrr8k\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-kube-api-access-nrr8k\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999042 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-podinfo\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999103 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10e6edf-82fb-4819-aa04-21f0db05769b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e6edf-82fb-4819-aa04-21f0db05769b-istiod-ca-cert\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999162 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-envoy\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999281 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-data\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999323 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-certs" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:38.999355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999339 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-credential-socket\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999423 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-socket\") pod \"c10e6edf-82fb-4819-aa04-21f0db05769b\" (UID: \"c10e6edf-82fb-4819-aa04-21f0db05769b\") " Oct 31 15:12:38.999610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999595 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-data" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:38.999724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999683 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-credential-socket" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:38.999786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999720 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10e6edf-82fb-4819-aa04-21f0db05769b-istiod-ca-cert" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:38.999912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:38.999854 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-socket" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.000195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.000177 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.000266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.000198 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.000266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.000214 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10e6edf-82fb-4819-aa04-21f0db05769b-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.000266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.000228 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.000266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.000241 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.001650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.001627 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-envoy" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.001650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.001630 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-token" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.002102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.002084 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-podinfo" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:39.002184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.002139 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-kube-api-access-nrr8k" pod "c10e6edf-82fb-4819-aa04-21f0db05769b" (UID: "c10e6edf-82fb-4819-aa04-21f0db05769b"). InnerVolumeSpecName "kube-api-access-nrr8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.101843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.101802 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.101843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.101838 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrr8k\" (UniqueName: \"kubernetes.io/projected/c10e6edf-82fb-4819-aa04-21f0db05769b-kube-api-access-nrr8k\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.102034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.101852 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.102034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.101865 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10e6edf-82fb-4819-aa04-21f0db05769b-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.391276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.391090 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg"] Oct 31 15:12:39.396785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:39.396756 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-9m8gg"] Oct 31 15:12:40.341679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:40.341646 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:40.594542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:40.588308 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" path="/var/lib/kubelet/pods/c10e6edf-82fb-4819-aa04-21f0db05769b/volumes" Oct 31 15:12:40.705493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:40.702524 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:12:41.951569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:41.951526 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:42.090035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:42.089993 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:43.540066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:43.540034 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:44.069631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:44.069596 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw"] Oct 31 15:12:44.069923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:44.069842 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" containerID="cri-o://67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156" gracePeriod=30 Oct 31 15:12:44.076357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:44.076328 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:44.833961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:44.833929 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:47.148320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:47.148296 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:47.302493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:47.300538 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:47.437545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:47.437314 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:47.492352 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:12:47.492318 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:12:49.410709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.410682 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:12:49.422800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.422769 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istiod-ca-cert\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.422956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.422821 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-credential-socket\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.422956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.422859 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-podinfo\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.422956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.422931 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-envoy\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.423197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.422999 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-token\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-token\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.423197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423049 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-certs\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.423197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423079 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-kube-api-access-hjkrg\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-kube-api-access-hjkrg\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.423197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423105 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-credential-socket" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:49.423197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423140 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-socket\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.423197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423156 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istiod-ca-cert" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:49.423531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423220 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-data\") pod \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\" (UID: \"2e890f84-dc95-41f4-9b65-7ce6b635e5a6\") " Oct 31 15:12:49.423531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423336 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-certs" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:49.423637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423556 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-socket" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:49.423637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.423605 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-data" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:49.424219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.424192 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.424329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.424224 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.424329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.424240 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.424329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.424255 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.424329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.424271 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.425514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.425448 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-envoy" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:49.425514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.425457 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-podinfo" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:49.425681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.425663 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-token" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:49.425772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.425753 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-kube-api-access-hjkrg" pod "2e890f84-dc95-41f4-9b65-7ce6b635e5a6" (UID: "2e890f84-dc95-41f4-9b65-7ce6b635e5a6"). InnerVolumeSpecName "kube-api-access-hjkrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:49.525591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.525559 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjkrg\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-kube-api-access-hjkrg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.525591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.525586 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.525591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.525596 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:49.525839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:49.525605 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e890f84-dc95-41f4-9b65-7ce6b635e5a6-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:50.083952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.083914 4865 generic.go:356] "Generic (PLEG): container finished" podID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerID="67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156" exitCode=0 Oct 31 15:12:50.084168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.083975 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" Oct 31 15:12:50.084168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.083997 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" event={"ID":"2e890f84-dc95-41f4-9b65-7ce6b635e5a6","Type":"ContainerDied","Data":"67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156"} Oct 31 15:12:50.084168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.084055 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw" event={"ID":"2e890f84-dc95-41f4-9b65-7ce6b635e5a6","Type":"ContainerDied","Data":"54dc999db8e88df1b8bf555b0aaf04a6ad633d4942a139b04c46423210156bc3"} Oct 31 15:12:50.084168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.084079 4865 scope.go:122] "RemoveContainer" containerID="67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156" Oct 31 15:12:50.095374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.095353 4865 scope.go:122] "RemoveContainer" containerID="d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337" Oct 31 15:12:50.105559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.105544 4865 scope.go:122] "RemoveContainer" containerID="67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156" Oct 31 15:12:50.105855 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:12:50.105837 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156\": container with ID starting with 67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156 not found: ID does not exist" containerID="67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156" Oct 31 15:12:50.105901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.105864 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156"} err="failed to get container status \"67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156\": rpc error: code = NotFound desc = could not find container \"67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156\": container with ID starting with 67203e7c57f5933b173c4fe0bbb4605f3f73f1d7cc31ef5ba497ccd69cdf9156 not found: ID does not exist" Oct 31 15:12:50.105901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.105881 4865 scope.go:122] "RemoveContainer" containerID="d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337" Oct 31 15:12:50.106133 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:12:50.106118 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337\": container with ID starting with d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337 not found: ID does not exist" containerID="d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337" Oct 31 15:12:50.106194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.106136 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337"} err="failed to get container status \"d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337\": rpc error: code = NotFound desc = could not find container \"d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337\": container with ID starting with d8b1a42fb19f1e2ce1a4866d05b8d4cc4a9c63695b632c0bb0cf2495c72f5337 not found: ID does not exist" Oct 31 15:12:50.273494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.266751 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw"] Oct 31 15:12:50.324579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.324545 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4d3ea8-openshift-default-86f477c467-rfdnw"] Oct 31 15:12:50.333756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.333724 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz"] Oct 31 15:12:50.403906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.403825 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp"] Oct 31 15:12:50.523516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.522635 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" path="/var/lib/kubelet/pods/2e890f84-dc95-41f4-9b65-7ce6b635e5a6/volumes" Oct 31 15:12:50.646927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.646892 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:50.786435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.786396 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:50.914452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:50.914334 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:51.933353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:51.933254 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:53.762418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:53.762387 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:53.882717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:53.882682 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:55.145389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.145358 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:55.334448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.334408 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" podUID="c71e955c-d595-435b-9702-9ebf6ef2fc70" containerName="payload-processing" containerID="cri-o://48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf" gracePeriod=25 Oct 31 15:12:55.404275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.404204 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" podUID="18a65fdb-497b-4152-938f-854f35e3b818" containerName="payload-pre-processing" containerID="cri-o://25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465" gracePeriod=25 Oct 31 15:12:55.598271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.598140 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:55.698736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.698679 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:12:55.721647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.721621 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:55.724451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.724428 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:12:55.800539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.800500 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71e955c-d595-435b-9702-9ebf6ef2fc70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71e955c-d595-435b-9702-9ebf6ef2fc70-plugins-config-volume\") pod \"c71e955c-d595-435b-9702-9ebf6ef2fc70\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " Oct 31 15:12:55.800685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.800655 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a65fdb-497b-4152-938f-854f35e3b818-kube-api-access-mmsbw\" (UniqueName: \"kubernetes.io/projected/18a65fdb-497b-4152-938f-854f35e3b818-kube-api-access-mmsbw\") pod \"18a65fdb-497b-4152-938f-854f35e3b818\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " Oct 31 15:12:55.800767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.800752 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a65fdb-497b-4152-938f-854f35e3b818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a65fdb-497b-4152-938f-854f35e3b818-plugins-config-volume\") pod \"18a65fdb-497b-4152-938f-854f35e3b818\" (UID: \"18a65fdb-497b-4152-938f-854f35e3b818\") " Oct 31 15:12:55.800830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.800805 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71e955c-d595-435b-9702-9ebf6ef2fc70-kube-api-access-9t6hq\" (UniqueName: \"kubernetes.io/projected/c71e955c-d595-435b-9702-9ebf6ef2fc70-kube-api-access-9t6hq\") pod \"c71e955c-d595-435b-9702-9ebf6ef2fc70\" (UID: \"c71e955c-d595-435b-9702-9ebf6ef2fc70\") " Oct 31 15:12:55.800892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.800811 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71e955c-d595-435b-9702-9ebf6ef2fc70-plugins-config-volume" pod "c71e955c-d595-435b-9702-9ebf6ef2fc70" (UID: "c71e955c-d595-435b-9702-9ebf6ef2fc70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:55.801194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.801167 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a65fdb-497b-4152-938f-854f35e3b818-plugins-config-volume" pod "18a65fdb-497b-4152-938f-854f35e3b818" (UID: "18a65fdb-497b-4152-938f-854f35e3b818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:55.801437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.801402 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a65fdb-497b-4152-938f-854f35e3b818-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:55.801437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.801426 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c71e955c-d595-435b-9702-9ebf6ef2fc70-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:55.802955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.802932 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a65fdb-497b-4152-938f-854f35e3b818-kube-api-access-mmsbw" pod "18a65fdb-497b-4152-938f-854f35e3b818" (UID: "18a65fdb-497b-4152-938f-854f35e3b818"). InnerVolumeSpecName "kube-api-access-mmsbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:55.803149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.803130 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71e955c-d595-435b-9702-9ebf6ef2fc70-kube-api-access-9t6hq" pod "c71e955c-d595-435b-9702-9ebf6ef2fc70" (UID: "c71e955c-d595-435b-9702-9ebf6ef2fc70"). InnerVolumeSpecName "kube-api-access-9t6hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:55.902654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.902614 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6hq\" (UniqueName: \"kubernetes.io/projected/c71e955c-d595-435b-9702-9ebf6ef2fc70-kube-api-access-9t6hq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:55.902654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:55.902647 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmsbw\" (UniqueName: \"kubernetes.io/projected/18a65fdb-497b-4152-938f-854f35e3b818-kube-api-access-mmsbw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:56.203166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.203136 4865 generic.go:356] "Generic (PLEG): container finished" podID="c71e955c-d595-435b-9702-9ebf6ef2fc70" containerID="48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf" exitCode=0 Oct 31 15:12:56.203666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.203177 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" Oct 31 15:12:56.203666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.203218 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" event={"ID":"c71e955c-d595-435b-9702-9ebf6ef2fc70","Type":"ContainerDied","Data":"48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf"} Oct 31 15:12:56.203666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.203298 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz" event={"ID":"c71e955c-d595-435b-9702-9ebf6ef2fc70","Type":"ContainerDied","Data":"8eb331e6c5c93b4163658951eb4f0a900f6b5ee1035f24fb0e6823a707e789a3"} Oct 31 15:12:56.203666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.203322 4865 scope.go:122] "RemoveContainer" containerID="48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf" Oct 31 15:12:56.204813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.204790 4865 generic.go:356] "Generic (PLEG): container finished" podID="18a65fdb-497b-4152-938f-854f35e3b818" containerID="25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465" exitCode=0 Oct 31 15:12:56.204959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.204838 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" Oct 31 15:12:56.204959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.204876 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" event={"ID":"18a65fdb-497b-4152-938f-854f35e3b818","Type":"ContainerDied","Data":"25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465"} Oct 31 15:12:56.205084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.204960 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp" event={"ID":"18a65fdb-497b-4152-938f-854f35e3b818","Type":"ContainerDied","Data":"3de133d12b9af6d1a9492e3ae469aca2565701f92bdada7340cb86d158accdcf"} Oct 31 15:12:56.214428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.214404 4865 scope.go:122] "RemoveContainer" containerID="48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf" Oct 31 15:12:56.214838 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:12:56.214735 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf\": container with ID starting with 48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf not found: ID does not exist" containerID="48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf" Oct 31 15:12:56.214838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.214771 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf"} err="failed to get container status \"48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf\": rpc error: code = NotFound desc = could not find container \"48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf\": container with ID starting with 48042a24f3275e9d69ea22f2f57a5e64c4a444647108ab96be5a66f13026c1cf not found: ID does not exist" Oct 31 15:12:56.214838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.214808 4865 scope.go:122] "RemoveContainer" containerID="25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465" Oct 31 15:12:56.224168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.224151 4865 scope.go:122] "RemoveContainer" containerID="25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465" Oct 31 15:12:56.224417 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:12:56.224398 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465\": container with ID starting with 25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465 not found: ID does not exist" containerID="25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465" Oct 31 15:12:56.224520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.224428 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465"} err="failed to get container status \"25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465\": rpc error: code = NotFound desc = could not find container \"25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465\": container with ID starting with 25468a5c9a3bf4d642580db8df31c554a5e82f0f99d956a524c10173972c8465 not found: ID does not exist" Oct 31 15:12:56.281094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.281071 4865 scope.go:122] "RemoveContainer" containerID="4590a92f034df34809ecfc5be1bbd27300380402dc61864287159f9459843a44" Oct 31 15:12:56.290577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.290561 4865 scope.go:122] "RemoveContainer" containerID="7b5e2178959d0e626d20af9668d3482d988920531f2cd047b72865d1bb4b78e3" Oct 31 15:12:56.299255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.299236 4865 scope.go:122] "RemoveContainer" containerID="75b8ad36f1e7aee35cd41056773a28ed2f97c2b5d2861ada21426b9abb598382" Oct 31 15:12:56.308137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.308121 4865 scope.go:122] "RemoveContainer" containerID="5955cff87e711947a8b5ed1633adddee7d607e51f2bab4505f1f3422f3a6a803" Oct 31 15:12:56.316811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.316796 4865 scope.go:122] "RemoveContainer" containerID="907a3436be80713c4d1a96c75e0ca5628c4f1f6c1030659db9119730b3d57c3a" Oct 31 15:12:56.373745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.373018 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz"] Oct 31 15:12:56.399784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.399762 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:12:56.422308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.422283 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f17a1-669d67558c-lfdtz"] Oct 31 15:12:56.486513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.486490 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp"] Oct 31 15:12:56.498982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.498961 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71e955c-d595-435b-9702-9ebf6ef2fc70" path="/var/lib/kubelet/pods/c71e955c-d595-435b-9702-9ebf6ef2fc70/volumes" Oct 31 15:12:56.499288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:56.499275 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f17a1-648bbfcf75-qfxhp"] Oct 31 15:12:58.499331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:58.499301 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a65fdb-497b-4152-938f-854f35e3b818" path="/var/lib/kubelet/pods/18a65fdb-497b-4152-938f-854f35e3b818/volumes" Oct 31 15:12:58.708511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:12:58.708444 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:03.379811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:03.379782 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:03.541906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:03.541868 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:03.671837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:03.671750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:08.501897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:08.501864 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:11.048532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.048498 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8"] Oct 31 15:13:11.098947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.098916 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d617cbf-r55c8"] Oct 31 15:13:11.407250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.407150 4865 generic.go:356] "Generic (PLEG): container finished" podID="a00341e7-e710-4156-8c80-6947b6d581e7" containerID="bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a" exitCode=0 Oct 31 15:13:11.407250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.407229 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" event={"ID":"a00341e7-e710-4156-8c80-6947b6d581e7","Type":"ContainerDied","Data":"bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a"} Oct 31 15:13:11.407692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.407672 4865 scope.go:122] "RemoveContainer" containerID="bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a" Oct 31 15:13:11.409214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.409184 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:11.409267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.409255 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:11.409308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:11.409286 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:12.058336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:12.058307 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:12.482054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:12.482022 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" event={"ID":"a00341e7-e710-4156-8c80-6947b6d581e7","Type":"ContainerStarted","Data":"108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c"} Oct 31 15:13:12.529271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:12.529240 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8baaa68f-1047-46ce-9d45-7536bad3074c" path="/var/lib/kubelet/pods/8baaa68f-1047-46ce-9d45-7536bad3074c/volumes" Oct 31 15:13:12.557403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:12.557373 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:12.575427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:12.575401 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:13:12.575758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:12.575738 4865 patch_prober.go:29] interesting pod/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 15:13:12.575808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:12.575785 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 15:13:13.575663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:13.575626 4865 patch_prober.go:29] interesting pod/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 15:13:13.576063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:13.575688 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 15:13:14.459643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.459612 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5"] Oct 31 15:13:14.460255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460239 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e9cb6015-17c8-4476-9a88-4d7a2d693c4c" containerName="payload-processing" Oct 31 15:13:14.460301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460256 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cb6015-17c8-4476-9a88-4d7a2d693c4c" containerName="payload-processing" Oct 31 15:13:14.460301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460279 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" Oct 31 15:13:14.460301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460284 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460302 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460306 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460313 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460317 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460330 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c71e955c-d595-435b-9702-9ebf6ef2fc70" containerName="payload-processing" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460335 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71e955c-d595-435b-9702-9ebf6ef2fc70" containerName="payload-processing" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460342 4865 cpu_manager.go:414] "containerMap: removing container" podUID="14e7cede-ede5-4f4a-94d3-3af7ed7af956" containerName="payload-pre-processing" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460347 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e7cede-ede5-4f4a-94d3-3af7ed7af956" containerName="payload-pre-processing" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460362 4865 cpu_manager.go:414] "containerMap: removing container" podUID="18a65fdb-497b-4152-938f-854f35e3b818" containerName="payload-pre-processing" Oct 31 15:13:14.460397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460366 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a65fdb-497b-4152-938f-854f35e3b818" containerName="payload-pre-processing" Oct 31 15:13:14.460740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460460 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10e6edf-82fb-4819-aa04-21f0db05769b" containerName="istio-proxy" Oct 31 15:13:14.460740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460481 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" Oct 31 15:13:14.460740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460489 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71e955c-d595-435b-9702-9ebf6ef2fc70" containerName="payload-processing" Oct 31 15:13:14.460740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460495 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e7cede-ede5-4f4a-94d3-3af7ed7af956" containerName="payload-pre-processing" Oct 31 15:13:14.460740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460504 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a65fdb-497b-4152-938f-854f35e3b818" containerName="payload-pre-processing" Oct 31 15:13:14.460740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.460512 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cb6015-17c8-4476-9a88-4d7a2d693c4c" containerName="payload-processing" Oct 31 15:13:14.463953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.463932 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:14.527334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.527303 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5"] Oct 31 15:13:14.538350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.538323 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9474b299-b315-4569-b444-801ff9376570-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:14.538543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.538375 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2x4m\" (UniqueName: \"kubernetes.io/projected/9474b299-b315-4569-b444-801ff9376570-kube-api-access-z2x4m\") pod \"payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:14.571260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.571227 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-644a8a\"" Oct 31 15:13:14.571404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.571302 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-644a8a-dockercfg-j6pzf\"" Oct 31 15:13:14.575581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.575557 4865 patch_prober.go:29] interesting pod/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 15:13:14.575705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.575605 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 15:13:14.641416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.641382 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9474b299-b315-4569-b444-801ff9376570-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:14.641710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.641689 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2x4m\" (UniqueName: \"kubernetes.io/projected/9474b299-b315-4569-b444-801ff9376570-kube-api-access-z2x4m\") pod \"payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:14.645184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.645153 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9474b299-b315-4569-b444-801ff9376570-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:14.719548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.719429 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2x4m\" (UniqueName: \"kubernetes.io/projected/9474b299-b315-4569-b444-801ff9376570-kube-api-access-z2x4m\") pod \"payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:14.778126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:14.778098 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:15.120918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:15.120823 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:15.206551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:15.206507 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5"] Oct 31 15:13:15.268749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:15.265977 4865 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:13:15.533444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:15.533409 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" event={"ID":"9474b299-b315-4569-b444-801ff9376570","Type":"ContainerStarted","Data":"40fd19a9914530e05b185170f9c041f351e7d850d44cc83cfe8c26fb927443c3"} Oct 31 15:13:15.576066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:15.576030 4865 patch_prober.go:29] interesting pod/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 15:13:15.576580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:15.576088 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 15:13:16.576329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:16.576291 4865 patch_prober.go:29] interesting pod/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 15:13:16.576882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:16.576365 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 15:13:17.513253 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:13:17.513179 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:13:17.575621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:17.575586 4865 patch_prober.go:29] interesting pod/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 15:13:17.575823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:17.575644 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 15:13:18.559832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:18.559795 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" event={"ID":"9474b299-b315-4569-b444-801ff9376570","Type":"ContainerStarted","Data":"0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4"} Oct 31 15:13:18.560283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:18.559872 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:18.581617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:18.581590 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:13:18.581876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:18.581858 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:13:18.582572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:18.582553 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:13:18.679592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:18.679544 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" podStartSLOduration=2.15056445 podStartE2EDuration="4.679530097s" totalImagesPullingTime="2.528965647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:18.675654528 +0000 UTC m=+1644.295205945" watchObservedRunningTime="2026-10-31 15:13:18.679530097 +0000 UTC m=+1644.299081511" Oct 31 15:13:19.773239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.773208 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm"] Oct 31 15:13:19.774059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.774042 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e890f84-dc95-41f4-9b65-7ce6b635e5a6" containerName="istio-proxy" Oct 31 15:13:19.777728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.777713 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798360 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106bab7b-8945-48ee-96bb-0343910ef473-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798505 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798561 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798580 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl89f\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-kube-api-access-rl89f\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798651 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798677 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798697 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798734 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106bab7b-8945-48ee-96bb-0343910ef473-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.798938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.798917 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900102 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106bab7b-8945-48ee-96bb-0343910ef473-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900239 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900386 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900415 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl89f\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-kube-api-access-rl89f\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900460 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900518 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900548 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900606 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106bab7b-8945-48ee-96bb-0343910ef473-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900663 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.900770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900679 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.901004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.900982 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.901435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.901267 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.901592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.901298 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.901870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.901849 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106bab7b-8945-48ee-96bb-0343910ef473-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.903521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.903500 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106bab7b-8945-48ee-96bb-0343910ef473-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.903813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.903796 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:19.912738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:19.912716 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:20.036534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.027240 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm"] Oct 31 15:13:20.036534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.028617 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:20.036534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.035076 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl89f\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-kube-api-access-rl89f\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:20.092833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.092805 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:20.523352 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:13:20.523315 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106bab7b_8945_48ee_96bb_0343910ef473.slice/crio-1172a582aac0c9a78b36f7964827734d71e9887d8c3c439243cf94469f8dad71 WatchSource:0}: Error finding container 1172a582aac0c9a78b36f7964827734d71e9887d8c3c439243cf94469f8dad71: Status 404 returned error can't find the container with id 1172a582aac0c9a78b36f7964827734d71e9887d8c3c439243cf94469f8dad71 Oct 31 15:13:20.524404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.524353 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm"] Oct 31 15:13:20.526085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.526051 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:20.526168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.526117 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:20.526168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.526161 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:20.575792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:20.575769 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" event={"ID":"106bab7b-8945-48ee-96bb-0343910ef473","Type":"ContainerStarted","Data":"1172a582aac0c9a78b36f7964827734d71e9887d8c3c439243cf94469f8dad71"} Oct 31 15:13:21.046993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:21.046955 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm"] Oct 31 15:13:21.097637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:21.097598 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54130add-nk9zm"] Oct 31 15:13:21.623490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:21.622631 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" event={"ID":"106bab7b-8945-48ee-96bb-0343910ef473","Type":"ContainerStarted","Data":"7674b61a0465765297e7292ba19b29888796d17a75da31ef5b693c505bb0fa7e"} Oct 31 15:13:21.822943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:21.822877 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podStartSLOduration=2.8228551509999997 podStartE2EDuration="2.822855151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:21.79312901 +0000 UTC m=+1647.412680420" watchObservedRunningTime="2026-10-31 15:13:21.822855151 +0000 UTC m=+1647.442406564" Oct 31 15:13:22.103495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:22.098408 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:22.118754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:22.118679 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 15:13:22.118978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:22.118777 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 15:13:22.522010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:22.521973 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ccf549-27f3-4945-941b-32cce06cc473" path="/var/lib/kubelet/pods/a9ccf549-27f3-4945-941b-32cce06cc473/volumes" Oct 31 15:13:23.093746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:23.093713 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 15:13:23.094053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:23.094020 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 15:13:24.100496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:24.093622 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 15:13:24.100496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:24.093681 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 15:13:25.074660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:25.074627 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:25.118872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:25.118837 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 15:13:25.119438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:25.119403 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 15:13:25.283426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:25.283392 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:25.417740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:25.417639 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:26.113630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:26.113591 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 15:13:26.113925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:26.113895 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 15:13:27.093552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:27.093513 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 15:13:27.093963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:27.093580 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 15:13:28.104610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:28.104579 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:28.119497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:28.113928 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:28.152357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:28.152319 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:13:28.683671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:28.683620 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:28.805704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:28.805667 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:13:29.568999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:29.568973 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:13:30.330702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:30.330669 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:32.870905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:32.870869 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:33.805344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:33.805308 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:34.047126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:34.047091 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc"] Oct 31 15:13:34.121047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:34.120958 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b48eec-6lwkc"] Oct 31 15:13:34.535743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:34.535707 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb30a083-1a19-4d31-8189-2d57471e14b5" path="/var/lib/kubelet/pods/cb30a083-1a19-4d31-8189-2d57471e14b5/volumes" Oct 31 15:13:34.921915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:34.921827 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4"] Oct 31 15:13:34.928031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:34.928013 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.042780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.042740 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4"] Oct 31 15:13:35.076256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076216 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076270 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076299 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076551 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076665 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076719 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076765 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076833 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.076964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.076915 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4f7b\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-kube-api-access-j4f7b\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.105220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.105187 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:35.195603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.195367 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.204402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.204368 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.223605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.223571 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.224106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.224059 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f7b\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-kube-api-access-j4f7b\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.224504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.224447 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.224729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.224686 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.224915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.224880 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.226286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.226236 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.227693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.196099 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.227847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.227039 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.241530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.227498 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.241789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.227607 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.241936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.231092 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.242389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.242358 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.250004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.249693 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.250004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.249010 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.299911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.299880 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.300056 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.299964 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f7b\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-kube-api-access-j4f7b\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-clcd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.543080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.543050 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:35.980613 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:13:35.980579 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa520adc_b9e5_42a1_b1b2_c487a73fedd4.slice/crio-ecb1b9448fe8d5c0df0dd2ed8b0ec2aa26ca11fa170a7323fcfe0a835654d53e WatchSource:0}: Error finding container ecb1b9448fe8d5c0df0dd2ed8b0ec2aa26ca11fa170a7323fcfe0a835654d53e: Status 404 returned error can't find the container with id ecb1b9448fe8d5c0df0dd2ed8b0ec2aa26ca11fa170a7323fcfe0a835654d53e Oct 31 15:13:35.981176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.981129 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4"] Oct 31 15:13:35.983021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.982985 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:35.983103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.983062 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:35.983143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:35.983104 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:36.817493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:36.816808 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" event={"ID":"aa520adc-b9e5-42a1-b1b2-c487a73fedd4","Type":"ContainerStarted","Data":"874302466e17ce8dc4c2f0216ea735f58f766d2538b41132c633d2aca2e74ba9"} Oct 31 15:13:36.817493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:36.816875 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" event={"ID":"aa520adc-b9e5-42a1-b1b2-c487a73fedd4","Type":"ContainerStarted","Data":"ecb1b9448fe8d5c0df0dd2ed8b0ec2aa26ca11fa170a7323fcfe0a835654d53e"} Oct 31 15:13:37.038421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:37.038365 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podStartSLOduration=3.038348473 podStartE2EDuration="3.038348473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:36.986511794 +0000 UTC m=+1662.606063205" watchObservedRunningTime="2026-10-31 15:13:37.038348473 +0000 UTC m=+1662.657899885" Oct 31 15:13:37.544124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:37.544092 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:37.545741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:37.545715 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 15:13:37.545865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:37.545772 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 15:13:38.472616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:38.472571 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:13:38.472991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:38.472638 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:13:38.544191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:38.544157 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 15:13:38.544365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:38.544206 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 15:13:39.544097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.544062 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 15:13:39.544593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.544121 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 15:13:39.838608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.838518 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:39.845649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.845621 4865 generic.go:356] "Generic (PLEG): container finished" podID="e480e53c-7fff-4aad-92f7-c52755b17294" containerID="1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec" exitCode=0 Oct 31 15:13:39.845808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.845681 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" event={"ID":"e480e53c-7fff-4aad-92f7-c52755b17294","Type":"ContainerDied","Data":"1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec"} Oct 31 15:13:39.846146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.846127 4865 scope.go:122] "RemoveContainer" containerID="1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec" Oct 31 15:13:39.847759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.847730 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:39.847818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.847796 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:39.847859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:39.847834 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:40.543961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:40.543931 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 15:13:40.544167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:40.543986 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 15:13:40.855299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:40.855211 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" event={"ID":"e480e53c-7fff-4aad-92f7-c52755b17294","Type":"ContainerStarted","Data":"4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca"} Oct 31 15:13:41.328493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:41.325854 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:13:41.328493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:41.325990 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 15:13:41.328493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:41.326035 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 15:13:41.544496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:41.543860 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 15:13:41.544496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:41.543924 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 15:13:42.319869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:42.319828 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 15:13:42.320069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:42.319896 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 15:13:42.767213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:42.767165 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:42.768422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:42.768396 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:42.769017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:42.769002 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:13:43.320329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:43.320300 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 15:13:43.320549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:43.320355 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 15:13:44.319806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:44.319767 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 15:13:44.320234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:44.319824 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 15:13:44.619937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:44.619843 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:45.319836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:45.319799 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 15:13:45.320229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:45.319883 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 15:13:46.319948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:46.319914 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 31 15:13:46.320512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:46.319979 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 31 15:13:47.370502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:47.370174 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:13:47.389492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:47.384842 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:13:47.407902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:47.407870 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:13:47.581068 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:13:47.580981 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:13:49.318540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:49.311437 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:54.086512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:54.086461 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:55.343580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:55.343102 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:55.529941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:55.529900 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:56.231937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:56.231897 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:56.410259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:56.410223 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:56.504354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:56.504286 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh"] Oct 31 15:13:56.504553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:56.504444 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" containerID="cri-o://aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492" gracePeriod=30 Oct 31 15:13:56.558877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:56.558843 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p"] Oct 31 15:13:56.559105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:56.559081 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" containerID="cri-o://6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c" gracePeriod=30 Oct 31 15:13:57.780098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:57.780070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:57.999385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:57.999350 4865 generic.go:356] "Generic (PLEG): container finished" podID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerID="f47a32101578a89e75362edbedcc883cca0aeb339c3f4a64e69d564b36859ed3" exitCode=0 Oct 31 15:13:57.999599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:57.999423 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" event={"ID":"f12211dd-e275-441c-8e43-088e4c2bc3af","Type":"ContainerDied","Data":"f47a32101578a89e75362edbedcc883cca0aeb339c3f4a64e69d564b36859ed3"} Oct 31 15:13:57.999916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:57.999896 4865 scope.go:122] "RemoveContainer" containerID="f47a32101578a89e75362edbedcc883cca0aeb339c3f4a64e69d564b36859ed3" Oct 31 15:13:58.001513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.001483 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:58.001585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.001554 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:58.001627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.001589 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:13:58.261690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.261655 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56"] Oct 31 15:13:58.261933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.261906 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" containerID="cri-o://48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e" gracePeriod=30 Oct 31 15:13:58.266343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.266318 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w"] Oct 31 15:13:58.267110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.266546 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" containerID="cri-o://141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37" gracePeriod=30 Oct 31 15:13:58.273185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.273163 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq"] Oct 31 15:13:58.273603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.273530 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" containerID="cri-o://35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e" gracePeriod=30 Oct 31 15:13:58.287430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:58.287401 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:59.009055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.009018 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" event={"ID":"f12211dd-e275-441c-8e43-088e4c2bc3af","Type":"ContainerStarted","Data":"fba018447d234b4b0a91b9b353e5b678767af6d454ccc8f201bda29356ca68b7"} Oct 31 15:13:59.119397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.119328 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:59.226969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.226931 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:13:59.227324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.227301 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 15:13:59.227523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.227498 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 15:13:59.516087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.516046 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:13:59.945548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.944640 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:13:59.945548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:13:59.944703 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:00.231870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:00.231835 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 15:14:00.232490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:00.232440 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 15:14:00.418682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:00.418650 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:00.583070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:00.582993 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:00.722841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:00.722806 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:00.877615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:00.877462 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:01.226943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.226907 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 15:14:01.227131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.226961 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 15:14:01.877799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.877763 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:14:01.930370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.930341 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970308 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/744f6109-6a92-428a-9567-a268d095ca81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744f6109-6a92-428a-9567-a268d095ca81-istio-podinfo\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970392 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-data\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970437 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-credential-socket\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970517 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744f6109-6a92-428a-9567-a268d095ca81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744f6109-6a92-428a-9567-a268d095ca81-istiod-ca-cert\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970598 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-envoy\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970633 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-certs\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970688 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-kube-api-access-c48dp\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-kube-api-access-c48dp\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970929 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-istio-token\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-istio-token\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.971074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.970987 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-socket\") pod \"744f6109-6a92-428a-9567-a268d095ca81\" (UID: \"744f6109-6a92-428a-9567-a268d095ca81\") " Oct 31 15:14:01.972427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.972400 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-socket" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:01.972621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.972602 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-data" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:01.972790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.972771 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-credential-socket" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:01.973051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.973033 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744f6109-6a92-428a-9567-a268d095ca81-istiod-ca-cert" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:01.975854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.975608 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/744f6109-6a92-428a-9567-a268d095ca81-istio-podinfo" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:01.976009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.975949 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-kube-api-access-c48dp" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "kube-api-access-c48dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:01.976187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.976160 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-certs" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:01.977522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.976362 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-envoy" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:01.979662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:01.978137 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-istio-token" pod "744f6109-6a92-428a-9567-a268d095ca81" (UID: "744f6109-6a92-428a-9567-a268d095ca81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:02.072908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.072827 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-kube-api-access-ss4hb\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-kube-api-access-ss4hb\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.072908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.072867 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-envoy\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.072908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.072906 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-certs\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.073211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.072982 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10516171-e5bf-4736-a327-3b00e55df65e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10516171-e5bf-4736-a327-3b00e55df65e-istio-podinfo\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.073211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073018 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-istio-token\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-istio-token\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.073211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073054 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-credential-socket\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.073376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073233 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-socket\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.073376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073280 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-data\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.073376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073341 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10516171-e5bf-4736-a327-3b00e55df65e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10516171-e5bf-4736-a327-3b00e55df65e-istiod-ca-cert\") pod \"10516171-e5bf-4736-a327-3b00e55df65e\" (UID: \"10516171-e5bf-4736-a327-3b00e55df65e\") " Oct 31 15:14:02.073701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073421 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-credential-socket" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:02.073701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073674 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-socket" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:02.073987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.073963 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10516171-e5bf-4736-a327-3b00e55df65e-istiod-ca-cert" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:02.074185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074137 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-data" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:02.074268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074209 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074230 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744f6109-6a92-428a-9567-a268d095ca81-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074245 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074260 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074276 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744f6109-6a92-428a-9567-a268d095ca81-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074306 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074320 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074334 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48dp\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-kube-api-access-c48dp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074350 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074365 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10516171-e5bf-4736-a327-3b00e55df65e-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074380 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744f6109-6a92-428a-9567-a268d095ca81-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074394 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744f6109-6a92-428a-9567-a268d095ca81-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.074494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.074423 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-certs" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:02.076083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.076053 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-envoy" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:02.076197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.076112 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-kube-api-access-ss4hb" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "kube-api-access-ss4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:02.076413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.076386 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-istio-token" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:02.076972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.076949 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10516171-e5bf-4736-a327-3b00e55df65e-istio-podinfo" pod "10516171-e5bf-4736-a327-3b00e55df65e" (UID: "10516171-e5bf-4736-a327-3b00e55df65e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:02.138337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.138300 4865 generic.go:356] "Generic (PLEG): container finished" podID="744f6109-6a92-428a-9567-a268d095ca81" containerID="aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492" exitCode=0 Oct 31 15:14:02.138541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.138369 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" Oct 31 15:14:02.138541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.138377 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" event={"ID":"744f6109-6a92-428a-9567-a268d095ca81","Type":"ContainerDied","Data":"aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492"} Oct 31 15:14:02.138541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.138453 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh" event={"ID":"744f6109-6a92-428a-9567-a268d095ca81","Type":"ContainerDied","Data":"95a220deda932cce757bf533ae5874e7b467357bbbf33ab381a43f1d77225123"} Oct 31 15:14:02.138541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.138494 4865 scope.go:122] "RemoveContainer" containerID="aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492" Oct 31 15:14:02.141361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.141330 4865 generic.go:356] "Generic (PLEG): container finished" podID="10516171-e5bf-4736-a327-3b00e55df65e" containerID="6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c" exitCode=0 Oct 31 15:14:02.141462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.141434 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" Oct 31 15:14:02.141538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.141461 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" event={"ID":"10516171-e5bf-4736-a327-3b00e55df65e","Type":"ContainerDied","Data":"6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c"} Oct 31 15:14:02.141597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.141533 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p" event={"ID":"10516171-e5bf-4736-a327-3b00e55df65e","Type":"ContainerDied","Data":"694f39bb8fa2f4567d5f8b52aee2047a7de25a443e330c162be0e4de61acf268"} Oct 31 15:14:02.152137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.152115 4865 scope.go:122] "RemoveContainer" containerID="aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492" Oct 31 15:14:02.152446 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:02.152424 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492\": container with ID starting with aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492 not found: ID does not exist" containerID="aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492" Oct 31 15:14:02.152569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.152456 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492"} err="failed to get container status \"aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492\": rpc error: code = NotFound desc = could not find container \"aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492\": container with ID starting with aef96a83c8a6493e6e995dc7c5a19e19807bb2e4a80a38762c3f2aa166b48492 not found: ID does not exist" Oct 31 15:14:02.152569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.152497 4865 scope.go:122] "RemoveContainer" containerID="6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c" Oct 31 15:14:02.177640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.177598 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.177640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.177643 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4hb\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-kube-api-access-ss4hb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.177850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.177661 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.177850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.177674 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10516171-e5bf-4736-a327-3b00e55df65e-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.177850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.177688 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10516171-e5bf-4736-a327-3b00e55df65e-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.177850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.177703 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10516171-e5bf-4736-a327-3b00e55df65e-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:02.179074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.179037 4865 scope.go:122] "RemoveContainer" containerID="6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c" Oct 31 15:14:02.179553 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:02.179529 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c\": container with ID starting with 6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c not found: ID does not exist" containerID="6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c" Oct 31 15:14:02.179675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.179559 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c"} err="failed to get container status \"6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c\": rpc error: code = NotFound desc = could not find container \"6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c\": container with ID starting with 6ffff0fc8e73d07ea3c121d2ac703e2b0dbccdc94983c46653aadd23f74fe55c not found: ID does not exist" Oct 31 15:14:02.227015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.226985 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 15:14:02.227189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.227035 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 15:14:02.326296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.325257 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p"] Oct 31 15:14:02.330929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.330897 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hgq7p"] Oct 31 15:14:02.421041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.421005 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh"] Oct 31 15:14:02.424520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.424497 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4gmkh"] Oct 31 15:14:02.565556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.559890 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:02.565556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.559946 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:02.565556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.561515 4865 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:02.565556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.561580 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:02.585583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.578213 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10516171-e5bf-4736-a327-3b00e55df65e" path="/var/lib/kubelet/pods/10516171-e5bf-4736-a327-3b00e55df65e/volumes" Oct 31 15:14:02.585583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:02.578983 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744f6109-6a92-428a-9567-a268d095ca81" path="/var/lib/kubelet/pods/744f6109-6a92-428a-9567-a268d095ca81/volumes" Oct 31 15:14:03.135925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.135895 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:14:03.226799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.226765 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 15:14:03.227001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.226817 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 15:14:03.665288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.665257 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:14:03.733856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.733657 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-kube-api-access-gv9mp\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-kube-api-access-gv9mp\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.733856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.733775 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1c0901-ef24-4b51-9244-829417ec99cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c0901-ef24-4b51-9244-829417ec99cb-istiod-ca-cert\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.733856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.733843 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-certs\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.734165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.733879 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-socket\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.734165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.733940 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-envoy\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.734165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.733972 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-credential-socket\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.734165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.734037 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-token\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.734165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.734094 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-podinfo\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.734384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.734279 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-data\") pod \"9e1c0901-ef24-4b51-9244-829417ec99cb\" (UID: \"9e1c0901-ef24-4b51-9244-829417ec99cb\") " Oct 31 15:14:03.735368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.734729 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-certs" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.735368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.734991 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-data" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.735368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.735064 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.735368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.735194 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-credential-socket" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.735885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.735848 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1c0901-ef24-4b51-9244-829417ec99cb-istiod-ca-cert" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:03.737073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.737020 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-socket" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.738077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.737346 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-podinfo" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:03.738934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.738897 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-envoy" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.739059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.738954 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-kube-api-access-gv9mp" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "kube-api-access-gv9mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.741208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.740974 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-token" pod "9e1c0901-ef24-4b51-9244-829417ec99cb" (UID: "9e1c0901-ef24-4b51-9244-829417ec99cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.783746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.783719 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:14:03.786801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.786781 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:14:03.836915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836885 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c0901-ef24-4b51-9244-829417ec99cb-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.836915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836913 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.836915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836924 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.837268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836934 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.837268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836942 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.837268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836950 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.837268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836960 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c0901-ef24-4b51-9244-829417ec99cb-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.837268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.836969 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9mp\" (UniqueName: \"kubernetes.io/projected/9e1c0901-ef24-4b51-9244-829417ec99cb-kube-api-access-gv9mp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.938436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938348 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-kube-api-access-59658\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-kube-api-access-59658\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.938436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938412 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-istio-token\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-istio-token\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938498 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-kube-api-access-5xml8\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-kube-api-access-5xml8\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938567 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-data\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938605 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf190734-87d5-4a23-87d1-91593a593fe4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf190734-87d5-4a23-87d1-91593a593fe4-istiod-ca-cert\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938654 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf190734-87d5-4a23-87d1-91593a593fe4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf190734-87d5-4a23-87d1-91593a593fe4-istio-podinfo\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938734 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-certs\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938772 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-socket\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938833 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-envoy\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.938926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938865 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-credential-socket\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.938926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938918 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76f344a2-7bef-4813-982b-e56c2a07e132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f344a2-7bef-4813-982b-e56c2a07e132-istio-podinfo\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.939196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938951 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-istio-token\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-istio-token\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.939196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938990 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-envoy\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.939196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939047 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f344a2-7bef-4813-982b-e56c2a07e132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f344a2-7bef-4813-982b-e56c2a07e132-istiod-ca-cert\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.939196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939074 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-certs\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.939196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939102 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-credential-socket\") pod \"cf190734-87d5-4a23-87d1-91593a593fe4\" (UID: \"cf190734-87d5-4a23-87d1-91593a593fe4\") " Oct 31 15:14:03.939196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.938896 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-data" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.939196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939187 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-data\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.939616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939274 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-certs" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.939616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939323 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-socket" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.939616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939360 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-socket\") pod \"76f344a2-7bef-4813-982b-e56c2a07e132\" (UID: \"76f344a2-7bef-4813-982b-e56c2a07e132\") " Oct 31 15:14:03.939616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939408 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-data" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.939827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.939754 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f344a2-7bef-4813-982b-e56c2a07e132-istiod-ca-cert" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:03.940051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940022 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-socket" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.940283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940257 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf190734-87d5-4a23-87d1-91593a593fe4-istiod-ca-cert" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:03.940458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940436 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.940547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940485 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.940547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940502 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.940547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940518 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf190734-87d5-4a23-87d1-91593a593fe4-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.940547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940534 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.940807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940547 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.940807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940561 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f344a2-7bef-4813-982b-e56c2a07e132-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.940807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940684 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-certs" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.940807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940744 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-credential-socket" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.941035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.940990 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-credential-socket" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.943386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.943355 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-kube-api-access-5xml8" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "kube-api-access-5xml8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.944889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.944851 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76f344a2-7bef-4813-982b-e56c2a07e132-istio-podinfo" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:03.945026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.945007 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-envoy" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.945164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.945130 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-istio-token" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.945570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.945549 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-envoy" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.945914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.945889 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-istio-token" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.946133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.946111 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-kube-api-access-59658" pod "76f344a2-7bef-4813-982b-e56c2a07e132" (UID: "76f344a2-7bef-4813-982b-e56c2a07e132"). InnerVolumeSpecName "kube-api-access-59658". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.948602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:03.948582 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf190734-87d5-4a23-87d1-91593a593fe4-istio-podinfo" pod "cf190734-87d5-4a23-87d1-91593a593fe4" (UID: "cf190734-87d5-4a23-87d1-91593a593fe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:04.042008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.041977 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59658\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-kube-api-access-59658\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042005 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042018 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xml8\" (UniqueName: \"kubernetes.io/projected/cf190734-87d5-4a23-87d1-91593a593fe4-kube-api-access-5xml8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042032 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf190734-87d5-4a23-87d1-91593a593fe4-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042046 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042057 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042069 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f344a2-7bef-4813-982b-e56c2a07e132-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042081 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f344a2-7bef-4813-982b-e56c2a07e132-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042092 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042104 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f344a2-7bef-4813-982b-e56c2a07e132-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.042238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.042117 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf190734-87d5-4a23-87d1-91593a593fe4-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.175026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.174973 4865 generic.go:356] "Generic (PLEG): container finished" podID="cf190734-87d5-4a23-87d1-91593a593fe4" containerID="35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e" exitCode=0 Oct 31 15:14:04.175523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.175077 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" Oct 31 15:14:04.175523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.175114 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" event={"ID":"cf190734-87d5-4a23-87d1-91593a593fe4","Type":"ContainerDied","Data":"35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e"} Oct 31 15:14:04.175523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.175349 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq" event={"ID":"cf190734-87d5-4a23-87d1-91593a593fe4","Type":"ContainerDied","Data":"5fcd0df4db08e10dd6c55798de26c5a7c78f8f870933d1a81533a3ba676ea537"} Oct 31 15:14:04.175523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.175389 4865 scope.go:122] "RemoveContainer" containerID="35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e" Oct 31 15:14:04.177201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.177075 4865 generic.go:356] "Generic (PLEG): container finished" podID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerID="48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e" exitCode=0 Oct 31 15:14:04.177201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.177106 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" event={"ID":"9e1c0901-ef24-4b51-9244-829417ec99cb","Type":"ContainerDied","Data":"48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e"} Oct 31 15:14:04.177201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.177166 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" Oct 31 15:14:04.177201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.177176 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56" event={"ID":"9e1c0901-ef24-4b51-9244-829417ec99cb","Type":"ContainerDied","Data":"20f26413b490eae30a2f4224849dbc134f1742aa715587062555b9d205a0a5f4"} Oct 31 15:14:04.179166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.179141 4865 generic.go:356] "Generic (PLEG): container finished" podID="76f344a2-7bef-4813-982b-e56c2a07e132" containerID="141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37" exitCode=0 Oct 31 15:14:04.179282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.179206 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" event={"ID":"76f344a2-7bef-4813-982b-e56c2a07e132","Type":"ContainerDied","Data":"141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37"} Oct 31 15:14:04.179282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.179236 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" event={"ID":"76f344a2-7bef-4813-982b-e56c2a07e132","Type":"ContainerDied","Data":"e7f4226996cf76edb90e54f99aa553518e6d1c74a73e8e7ce6484a46254ec3d9"} Oct 31 15:14:04.179394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.179313 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w" Oct 31 15:14:04.189021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.189007 4865 scope.go:122] "RemoveContainer" containerID="35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e" Oct 31 15:14:04.189310 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:04.189288 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e\": container with ID starting with 35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e not found: ID does not exist" containerID="35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e" Oct 31 15:14:04.189397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.189319 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e"} err="failed to get container status \"35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e\": rpc error: code = NotFound desc = could not find container \"35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e\": container with ID starting with 35de3bb14efe91a46aa261204e2584461e81f2f36d3ef507aad80d2d404c5a5e not found: ID does not exist" Oct 31 15:14:04.189397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.189341 4865 scope.go:122] "RemoveContainer" containerID="48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e" Oct 31 15:14:04.224610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.224585 4865 scope.go:122] "RemoveContainer" containerID="48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e" Oct 31 15:14:04.224979 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:04.224956 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e\": container with ID starting with 48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e not found: ID does not exist" containerID="48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e" Oct 31 15:14:04.225092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.224991 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e"} err="failed to get container status \"48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e\": rpc error: code = NotFound desc = could not find container \"48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e\": container with ID starting with 48d9053d9cc3441d37a1c7da151bd977301383278621e04a9e0b55c59497371e not found: ID does not exist" Oct 31 15:14:04.225092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.225036 4865 scope.go:122] "RemoveContainer" containerID="141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37" Oct 31 15:14:04.226883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.226859 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 15:14:04.226974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.226902 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 15:14:04.235370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.235349 4865 scope.go:122] "RemoveContainer" containerID="141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37" Oct 31 15:14:04.235673 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:04.235654 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37\": container with ID starting with 141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37 not found: ID does not exist" containerID="141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37" Oct 31 15:14:04.235730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.235684 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37"} err="failed to get container status \"141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37\": rpc error: code = NotFound desc = could not find container \"141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37\": container with ID starting with 141a7f3dc1e64c2f351aa508f632e2b478e8ee4ba8f89e1b563e9658aa9c9a37 not found: ID does not exist" Oct 31 15:14:04.344412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.344372 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56"] Oct 31 15:14:04.347766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.347747 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-49d56"] Oct 31 15:14:04.462451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.462418 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq"] Oct 31 15:14:04.466331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.466304 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-2wkfq"] Oct 31 15:14:04.517036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.517001 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" path="/var/lib/kubelet/pods/9e1c0901-ef24-4b51-9244-829417ec99cb/volumes" Oct 31 15:14:04.518748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.518724 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" path="/var/lib/kubelet/pods/cf190734-87d5-4a23-87d1-91593a593fe4/volumes" Oct 31 15:14:04.580909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.580873 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w"] Oct 31 15:14:04.584033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:04.584015 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-8vq5w"] Oct 31 15:14:05.116453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:05.116421 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:05.232038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:05.232008 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:14:05.232538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:05.232494 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:14:05.233162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:05.233143 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:14:05.293873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:05.293842 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:05.430894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:05.430810 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:06.498519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:06.498452 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" path="/var/lib/kubelet/pods/76f344a2-7bef-4813-982b-e56c2a07e132/volumes" Oct 31 15:14:06.587693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:06.587654 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:07.110132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:07.110097 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq"] Oct 31 15:14:07.227884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:07.227853 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc"] Oct 31 15:14:07.451997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:07.451916 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:07.614328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:07.604363 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:09.919604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:09.919512 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:10.651421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:10.651391 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:10.850564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:10.850531 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:11.104719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.104649 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:11.257033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.256998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:11.314693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.314659 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc"] Oct 31 15:14:11.316161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.316136 4865 cpu_manager.go:414] "containerMap: removing container" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" Oct 31 15:14:11.325007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.324983 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" Oct 31 15:14:11.325199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.325187 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" Oct 31 15:14:11.325740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.325721 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" Oct 31 15:14:11.325891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.325880 4865 cpu_manager.go:414] "containerMap: removing container" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" Oct 31 15:14:11.325984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.325972 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" Oct 31 15:14:11.326078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.326068 4865 cpu_manager.go:414] "containerMap: removing container" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" Oct 31 15:14:11.326939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.326922 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" Oct 31 15:14:11.327092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.327081 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" Oct 31 15:14:11.327176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.327166 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" Oct 31 15:14:11.327495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.327458 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f344a2-7bef-4813-982b-e56c2a07e132" containerName="istio-proxy" Oct 31 15:14:11.327622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.327611 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf190734-87d5-4a23-87d1-91593a593fe4" containerName="istio-proxy" Oct 31 15:14:11.327737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.327727 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="10516171-e5bf-4736-a327-3b00e55df65e" containerName="istio-proxy" Oct 31 15:14:11.327826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.327816 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1c0901-ef24-4b51-9244-829417ec99cb" containerName="istio-proxy" Oct 31 15:14:11.327968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.327957 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="744f6109-6a92-428a-9567-a268d095ca81" containerName="istio-proxy" Oct 31 15:14:11.332445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.332426 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:11.347615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.346552 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc"] Oct 31 15:14:11.395543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.392505 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-032b05-dockercfg-lhj6d\"" Oct 31 15:14:11.395543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.392604 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-032b05\"" Oct 31 15:14:11.522491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.505639 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws"] Oct 31 15:14:11.522491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.515008 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.538516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.525643 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843444d-dccd-41e0-b7e4-475e95ea4976-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:11.538516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.525867 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5bqz\" (UniqueName: \"kubernetes.io/projected/9843444d-dccd-41e0-b7e4-475e95ea4976-kube-api-access-m5bqz\") pod \"payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:11.582529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.577272 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws"] Oct 31 15:14:11.627128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.627091 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5bqz\" (UniqueName: \"kubernetes.io/projected/9843444d-dccd-41e0-b7e4-475e95ea4976-kube-api-access-m5bqz\") pod \"payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:11.627128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.627130 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2ee4de-20dc-4496-98e9-137cf6a2c853-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.627379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.627177 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843444d-dccd-41e0-b7e4-475e95ea4976-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:11.627379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.627248 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxq6\" (UniqueName: \"kubernetes.io/projected/6b2ee4de-20dc-4496-98e9-137cf6a2c853-kube-api-access-cfxq6\") pod \"payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.628133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.628108 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843444d-dccd-41e0-b7e4-475e95ea4976-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:11.687398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.687325 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5bqz\" (UniqueName: \"kubernetes.io/projected/9843444d-dccd-41e0-b7e4-475e95ea4976-kube-api-access-m5bqz\") pod \"payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:11.728454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.728426 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxq6\" (UniqueName: \"kubernetes.io/projected/6b2ee4de-20dc-4496-98e9-137cf6a2c853-kube-api-access-cfxq6\") pod \"payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.728631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.728602 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2ee4de-20dc-4496-98e9-137cf6a2c853-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.729194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.729170 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2ee4de-20dc-4496-98e9-137cf6a2c853-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.786614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.786579 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxq6\" (UniqueName: \"kubernetes.io/projected/6b2ee4de-20dc-4496-98e9-137cf6a2c853-kube-api-access-cfxq6\") pod \"payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.900699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.900667 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:11.970510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:11.967905 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:12.111292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.111254 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" podUID="ec856fce-255e-4afb-8a4f-546ae2b68ec7" containerName="payload-processing" containerID="cri-o://fae65899e447648e45de6706ffe815410c08596981b56fcd35550f339286c3e8" gracePeriod=25 Oct 31 15:14:12.228232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.228193 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" podUID="7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" containerName="payload-pre-processing" containerID="cri-o://ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576" gracePeriod=25 Oct 31 15:14:12.275397 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:14:12.275368 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2ee4de_20dc_4496_98e9_137cf6a2c853.slice/crio-f5b48d2534223b19de1fab987bcec6f4403671c62b88c99c83f1dacbad931c78 WatchSource:0}: Error finding container f5b48d2534223b19de1fab987bcec6f4403671c62b88c99c83f1dacbad931c78: Status 404 returned error can't find the container with id f5b48d2534223b19de1fab987bcec6f4403671c62b88c99c83f1dacbad931c78 Oct 31 15:14:12.276003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.275978 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws"] Oct 31 15:14:12.285096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.285065 4865 generic.go:356] "Generic (PLEG): container finished" podID="ec856fce-255e-4afb-8a4f-546ae2b68ec7" containerID="fae65899e447648e45de6706ffe815410c08596981b56fcd35550f339286c3e8" exitCode=0 Oct 31 15:14:12.285208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.285135 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" event={"ID":"ec856fce-255e-4afb-8a4f-546ae2b68ec7","Type":"ContainerDied","Data":"fae65899e447648e45de6706ffe815410c08596981b56fcd35550f339286c3e8"} Oct 31 15:14:12.386514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.386199 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc"] Oct 31 15:14:12.391530 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:14:12.391507 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9843444d_dccd_41e0_b7e4_475e95ea4976.slice/crio-643eb41131650c6d3d5d2fabe429f73226b2b189b1e7dc1ef33ffdc29647394e WatchSource:0}: Error finding container 643eb41131650c6d3d5d2fabe429f73226b2b189b1e7dc1ef33ffdc29647394e: Status 404 returned error can't find the container with id 643eb41131650c6d3d5d2fabe429f73226b2b189b1e7dc1ef33ffdc29647394e Oct 31 15:14:12.396818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.396800 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:14:12.438943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.437332 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec856fce-255e-4afb-8a4f-546ae2b68ec7-kube-api-access-m77gc\" (UniqueName: \"kubernetes.io/projected/ec856fce-255e-4afb-8a4f-546ae2b68ec7-kube-api-access-m77gc\") pod \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " Oct 31 15:14:12.438943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.437383 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec856fce-255e-4afb-8a4f-546ae2b68ec7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec856fce-255e-4afb-8a4f-546ae2b68ec7-plugins-config-volume\") pod \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\" (UID: \"ec856fce-255e-4afb-8a4f-546ae2b68ec7\") " Oct 31 15:14:12.438943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.437820 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec856fce-255e-4afb-8a4f-546ae2b68ec7-plugins-config-volume" pod "ec856fce-255e-4afb-8a4f-546ae2b68ec7" (UID: "ec856fce-255e-4afb-8a4f-546ae2b68ec7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:12.438943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.438734 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec856fce-255e-4afb-8a4f-546ae2b68ec7-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:12.440400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.440374 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec856fce-255e-4afb-8a4f-546ae2b68ec7-kube-api-access-m77gc" pod "ec856fce-255e-4afb-8a4f-546ae2b68ec7" (UID: "ec856fce-255e-4afb-8a4f-546ae2b68ec7"). InnerVolumeSpecName "kube-api-access-m77gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:12.539679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.539651 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m77gc\" (UniqueName: \"kubernetes.io/projected/ec856fce-255e-4afb-8a4f-546ae2b68ec7-kube-api-access-m77gc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:12.542695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.542670 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:14:12.640814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.640785 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-kube-api-access-gx2kj\" (UniqueName: \"kubernetes.io/projected/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-kube-api-access-gx2kj\") pod \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " Oct 31 15:14:12.640982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.640875 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-plugins-config-volume\") pod \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\" (UID: \"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba\") " Oct 31 15:14:12.641239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.641211 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-plugins-config-volume" pod "7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" (UID: "7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:12.641669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.641650 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:12.643190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.643166 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-kube-api-access-gx2kj" pod "7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" (UID: "7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba"). InnerVolumeSpecName "kube-api-access-gx2kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:12.743444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.743412 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2kj\" (UniqueName: \"kubernetes.io/projected/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba-kube-api-access-gx2kj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:12.821506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:12.821413 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:13.049248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.049215 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx"] Oct 31 15:14:13.052671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.052647 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93da8bef-d62qx"] Oct 31 15:14:13.295200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.295173 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" Oct 31 15:14:13.295715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.295210 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq" event={"ID":"ec856fce-255e-4afb-8a4f-546ae2b68ec7","Type":"ContainerDied","Data":"cfb50e266b5657c874ec953b3419927f0c8f8a055517bdd2b017e6cc70d6a31d"} Oct 31 15:14:13.295715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.295288 4865 scope.go:122] "RemoveContainer" containerID="fae65899e447648e45de6706ffe815410c08596981b56fcd35550f339286c3e8" Oct 31 15:14:13.298121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.298100 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" event={"ID":"9843444d-dccd-41e0-b7e4-475e95ea4976","Type":"ContainerStarted","Data":"51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d"} Oct 31 15:14:13.298258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.298141 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" event={"ID":"9843444d-dccd-41e0-b7e4-475e95ea4976","Type":"ContainerStarted","Data":"643eb41131650c6d3d5d2fabe429f73226b2b189b1e7dc1ef33ffdc29647394e"} Oct 31 15:14:13.298333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.298270 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:13.299519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.299496 4865 generic.go:356] "Generic (PLEG): container finished" podID="7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" containerID="ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576" exitCode=0 Oct 31 15:14:13.299643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.299527 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" Oct 31 15:14:13.299643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.299557 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" event={"ID":"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba","Type":"ContainerDied","Data":"ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576"} Oct 31 15:14:13.299643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.299597 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc" event={"ID":"7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba","Type":"ContainerDied","Data":"70cb5f79ef776cb578b3dd8ae4f9344ec846b2b792420cd0e152b3b5058ce492"} Oct 31 15:14:13.301341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.301322 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" event={"ID":"6b2ee4de-20dc-4496-98e9-137cf6a2c853","Type":"ContainerStarted","Data":"4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59"} Oct 31 15:14:13.301445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.301364 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" event={"ID":"6b2ee4de-20dc-4496-98e9-137cf6a2c853","Type":"ContainerStarted","Data":"f5b48d2534223b19de1fab987bcec6f4403671c62b88c99c83f1dacbad931c78"} Oct 31 15:14:13.301445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.301406 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:13.310726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.310708 4865 scope.go:122] "RemoveContainer" containerID="ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576" Oct 31 15:14:13.320215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.320198 4865 scope.go:122] "RemoveContainer" containerID="ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576" Oct 31 15:14:13.320481 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:13.320447 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576\": container with ID starting with ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576 not found: ID does not exist" containerID="ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576" Oct 31 15:14:13.320538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.320509 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576"} err="failed to get container status \"ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576\": rpc error: code = NotFound desc = could not find container \"ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576\": container with ID starting with ce63da0b2abe78fe69820083e61d800dfb7c68d29e4f748686b2ab70e146a576 not found: ID does not exist" Oct 31 15:14:13.462265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.462239 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc"] Oct 31 15:14:13.468704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.468680 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-aac61b-5f86cfb475-h4nvc"] Oct 31 15:14:13.578392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.578310 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq"] Oct 31 15:14:13.582047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.582016 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aac61b-6c89c447b7-25qrq"] Oct 31 15:14:13.695373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.695321 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" podStartSLOduration=2.695306945 podStartE2EDuration="2.695306945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:13.692708137 +0000 UTC m=+1699.312259547" watchObservedRunningTime="2026-10-31 15:14:13.695306945 +0000 UTC m=+1699.314858355" Oct 31 15:14:13.807871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:13.807801 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" podStartSLOduration=2.80777907 podStartE2EDuration="2.80777907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:13.807115473 +0000 UTC m=+1699.426666885" watchObservedRunningTime="2026-10-31 15:14:13.80777907 +0000 UTC m=+1699.427330483" Oct 31 15:14:14.256088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:14.256049 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq"] Oct 31 15:14:14.371142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:14.366544 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9"] Oct 31 15:14:14.497104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:14.497075 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" path="/var/lib/kubelet/pods/7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba/volumes" Oct 31 15:14:14.497542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:14.497527 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd05523f-56c9-49c2-a48f-496f5719e765" path="/var/lib/kubelet/pods/cd05523f-56c9-49c2-a48f-496f5719e765/volumes" Oct 31 15:14:14.497893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:14.497880 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec856fce-255e-4afb-8a4f-546ae2b68ec7" path="/var/lib/kubelet/pods/ec856fce-255e-4afb-8a4f-546ae2b68ec7/volumes" Oct 31 15:14:14.609405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:14.609330 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:14.771781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:14.769961 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:16.446183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:16.446140 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:16.866004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:16.865968 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:17.317548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:17.307243 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:17.503138 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:17.503105 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:14:18.167212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.167183 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:18.700010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.699976 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr"] Oct 31 15:14:18.700623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.700607 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ec856fce-255e-4afb-8a4f-546ae2b68ec7" containerName="payload-processing" Oct 31 15:14:18.700623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.700624 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec856fce-255e-4afb-8a4f-546ae2b68ec7" containerName="payload-processing" Oct 31 15:14:18.700759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.700660 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" containerName="payload-pre-processing" Oct 31 15:14:18.700759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.700665 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" containerName="payload-pre-processing" Oct 31 15:14:18.700870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.700775 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4b0d78-9a74-4fa3-80b9-d22ed2c0a4ba" containerName="payload-pre-processing" Oct 31 15:14:18.700870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.700786 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec856fce-255e-4afb-8a4f-546ae2b68ec7" containerName="payload-processing" Oct 31 15:14:18.704379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.704362 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:18.793740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.793702 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a538f1-openshift-default-dockercfg-gg52w\"" Oct 31 15:14:18.830012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:18.823070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.071492 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-socket\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.071613 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88xv\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-kube-api-access-s88xv\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.071785 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-podinfo\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.071855 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-data\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.072098 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-certs\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.072239 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-credential-socket\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.072411 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a588319-ac1c-4ff5-af1a-e7902e68b263-istiod-ca-cert\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.072691 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-envoy\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.074492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.072756 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-token\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.109592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.107615 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:19.111717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.111687 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr"] Oct 31 15:14:19.190439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.185024 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt"] Oct 31 15:14:19.200195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.200153 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f"] Oct 31 15:14:19.203708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.203681 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88xv\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-kube-api-access-s88xv\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.203834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.203736 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-podinfo\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.203834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.203785 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-data\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.203944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.203861 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-certs\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204238 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-credential-socket\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204276 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a588319-ac1c-4ff5-af1a-e7902e68b263-istiod-ca-cert\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204310 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-certs\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204341 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-data\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204374 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-envoy\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204420 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-token\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204511 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-socket\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204609 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-credential-socket\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.204937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.204917 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-socket\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.205074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.205054 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a588319-ac1c-4ff5-af1a-e7902e68b263-istiod-ca-cert\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.206485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.206447 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-podinfo\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.207885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.207864 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-envoy\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.262000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.261971 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88xv\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-kube-api-access-s88xv\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.262000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.261991 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-token\") pod \"e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.311183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.311134 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" podUID="1825c2c2-c391-4ab5-b8ac-3e06344a342d" containerName="payload-processing" containerID="cri-o://19946beba32880764e7416fc58de34810409d6a6e2b4726b2bb6048ad09e0448" gracePeriod=25 Oct 31 15:14:19.348350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.348317 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:19.376040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.370556 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" podUID="84c45d05-3186-46e6-993c-e3d87ac0b3d2" containerName="payload-pre-processing" containerID="cri-o://ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b" gracePeriod=25 Oct 31 15:14:19.436824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.436787 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:19.598573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.597429 4865 generic.go:356] "Generic (PLEG): container finished" podID="1825c2c2-c391-4ab5-b8ac-3e06344a342d" containerID="19946beba32880764e7416fc58de34810409d6a6e2b4726b2bb6048ad09e0448" exitCode=0 Oct 31 15:14:19.598573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.597744 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" event={"ID":"1825c2c2-c391-4ab5-b8ac-3e06344a342d","Type":"ContainerDied","Data":"19946beba32880764e7416fc58de34810409d6a6e2b4726b2bb6048ad09e0448"} Oct 31 15:14:19.598573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.598237 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:19.810755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.810723 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:14:19.859507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.859398 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1825c2c2-c391-4ab5-b8ac-3e06344a342d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825c2c2-c391-4ab5-b8ac-3e06344a342d-plugins-config-volume\") pod \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " Oct 31 15:14:19.859636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.859612 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1825c2c2-c391-4ab5-b8ac-3e06344a342d-kube-api-access-4khnr\" (UniqueName: \"kubernetes.io/projected/1825c2c2-c391-4ab5-b8ac-3e06344a342d-kube-api-access-4khnr\") pod \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\" (UID: \"1825c2c2-c391-4ab5-b8ac-3e06344a342d\") " Oct 31 15:14:19.859786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.859761 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1825c2c2-c391-4ab5-b8ac-3e06344a342d-plugins-config-volume" pod "1825c2c2-c391-4ab5-b8ac-3e06344a342d" (UID: "1825c2c2-c391-4ab5-b8ac-3e06344a342d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:19.860250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.860231 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1825c2c2-c391-4ab5-b8ac-3e06344a342d-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:19.862405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.862384 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1825c2c2-c391-4ab5-b8ac-3e06344a342d-kube-api-access-4khnr" pod "1825c2c2-c391-4ab5-b8ac-3e06344a342d" (UID: "1825c2c2-c391-4ab5-b8ac-3e06344a342d"). InnerVolumeSpecName "kube-api-access-4khnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:19.891096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.891075 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:14:19.961094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.961034 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c45d05-3186-46e6-993c-e3d87ac0b3d2-kube-api-access-xfs94\" (UniqueName: \"kubernetes.io/projected/84c45d05-3186-46e6-993c-e3d87ac0b3d2-kube-api-access-xfs94\") pod \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " Oct 31 15:14:19.961275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.961163 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c45d05-3186-46e6-993c-e3d87ac0b3d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c45d05-3186-46e6-993c-e3d87ac0b3d2-plugins-config-volume\") pod \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\" (UID: \"84c45d05-3186-46e6-993c-e3d87ac0b3d2\") " Oct 31 15:14:19.961562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.961542 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c45d05-3186-46e6-993c-e3d87ac0b3d2-plugins-config-volume" pod "84c45d05-3186-46e6-993c-e3d87ac0b3d2" (UID: "84c45d05-3186-46e6-993c-e3d87ac0b3d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:19.961706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.961692 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4khnr\" (UniqueName: \"kubernetes.io/projected/1825c2c2-c391-4ab5-b8ac-3e06344a342d-kube-api-access-4khnr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:19.961753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.961709 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84c45d05-3186-46e6-993c-e3d87ac0b3d2-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:19.963427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:19.963402 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c45d05-3186-46e6-993c-e3d87ac0b3d2-kube-api-access-xfs94" pod "84c45d05-3186-46e6-993c-e3d87ac0b3d2" (UID: "84c45d05-3186-46e6-993c-e3d87ac0b3d2"). InnerVolumeSpecName "kube-api-access-xfs94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:20.062928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.062892 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfs94\" (UniqueName: \"kubernetes.io/projected/84c45d05-3186-46e6-993c-e3d87ac0b3d2-kube-api-access-xfs94\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:20.095670 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:14:20.095639 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a588319_ac1c_4ff5_af1a_e7902e68b263.slice/crio-71fb2c457f88d6ff8d10e969a17f9246e6421c08d19147c50e0b081b682b2962 WatchSource:0}: Error finding container 71fb2c457f88d6ff8d10e969a17f9246e6421c08d19147c50e0b081b682b2962: Status 404 returned error can't find the container with id 71fb2c457f88d6ff8d10e969a17f9246e6421c08d19147c50e0b081b682b2962 Oct 31 15:14:20.095773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.095744 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr"] Oct 31 15:14:20.098112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.098081 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:14:20.098186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.098144 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:14:20.098186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.098172 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:14:20.639724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.638387 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" event={"ID":"9a588319-ac1c-4ff5-af1a-e7902e68b263","Type":"ContainerStarted","Data":"0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681"} Oct 31 15:14:20.639724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.638456 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" event={"ID":"9a588319-ac1c-4ff5-af1a-e7902e68b263","Type":"ContainerStarted","Data":"71fb2c457f88d6ff8d10e969a17f9246e6421c08d19147c50e0b081b682b2962"} Oct 31 15:14:20.641893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.641866 4865 generic.go:356] "Generic (PLEG): container finished" podID="84c45d05-3186-46e6-993c-e3d87ac0b3d2" containerID="ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b" exitCode=0 Oct 31 15:14:20.642420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.642351 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" event={"ID":"84c45d05-3186-46e6-993c-e3d87ac0b3d2","Type":"ContainerDied","Data":"ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b"} Oct 31 15:14:20.642685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.642626 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" event={"ID":"84c45d05-3186-46e6-993c-e3d87ac0b3d2","Type":"ContainerDied","Data":"f66369ec7be753f129b9101614d072d7217b12d862ed6ecd1395a1a61463c142"} Oct 31 15:14:20.652585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.642878 4865 scope.go:122] "RemoveContainer" containerID="ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b" Oct 31 15:14:20.652957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.646652 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9" Oct 31 15:14:20.659739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.658921 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" event={"ID":"1825c2c2-c391-4ab5-b8ac-3e06344a342d","Type":"ContainerDied","Data":"d9d9ea703af91274c32774f1348ab58bd692e992521f740d9e178ce6e47e9d48"} Oct 31 15:14:20.659739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.659030 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq" Oct 31 15:14:20.786536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.785111 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podStartSLOduration=2.785084591 podStartE2EDuration="2.785084591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:20.763147904 +0000 UTC m=+1706.382699316" watchObservedRunningTime="2026-10-31 15:14:20.785084591 +0000 UTC m=+1706.404636003" Oct 31 15:14:20.853050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.853024 4865 scope.go:122] "RemoveContainer" containerID="ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b" Oct 31 15:14:20.853767 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:20.853730 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b\": container with ID starting with ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b not found: ID does not exist" containerID="ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b" Oct 31 15:14:20.853853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.853766 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b"} err="failed to get container status \"ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b\": rpc error: code = NotFound desc = could not find container \"ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b\": container with ID starting with ce8875138f033b903d332cda30d93da418a7726ad1d4ac699fc2ac6615b5898b not found: ID does not exist" Oct 31 15:14:20.853853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:20.853786 4865 scope.go:122] "RemoveContainer" containerID="19946beba32880764e7416fc58de34810409d6a6e2b4726b2bb6048ad09e0448" Oct 31 15:14:21.020492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.015640 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9"] Oct 31 15:14:21.032494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.025686 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5695787b-648bfc6f79-5xsl9"] Oct 31 15:14:21.116818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.116651 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq"] Oct 31 15:14:21.121423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.121386 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5695787b-5c7448d59c-445hq"] Oct 31 15:14:21.372139 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.372070 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:21.377695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.377672 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:21.668569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.668450 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:21.669515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:21.669496 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:14:22.366698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:22.366669 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:22.499009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:22.498979 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1825c2c2-c391-4ab5-b8ac-3e06344a342d" path="/var/lib/kubelet/pods/1825c2c2-c391-4ab5-b8ac-3e06344a342d/volumes" Oct 31 15:14:22.499429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:22.499415 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c45d05-3186-46e6-993c-e3d87ac0b3d2" path="/var/lib/kubelet/pods/84c45d05-3186-46e6-993c-e3d87ac0b3d2/volumes" Oct 31 15:14:22.505880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:22.505802 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:24.003728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.003699 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:24.130997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.130970 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:24.204492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.198565 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" podUID="e0b53a6e-dcc2-468d-9b71-afab5b65dccf" containerName="payload-processing" containerID="cri-o://da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02" gracePeriod=25 Oct 31 15:14:24.204492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.201382 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" podUID="926c0093-7fa6-440e-afba-524657809b04" containerName="payload-pre-processing" containerID="cri-o://941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4" gracePeriod=25 Oct 31 15:14:24.313461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.313433 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:14:24.313646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.313545 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:14:24.536539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.526862 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:14:24.536539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.531160 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4"] Oct 31 15:14:24.536539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.531326 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" containerID="cri-o://874302466e17ce8dc4c2f0216ea735f58f766d2538b41132c633d2aca2e74ba9" gracePeriod=30 Oct 31 15:14:24.543140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.543110 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn"] Oct 31 15:14:24.543320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.543298 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" containerID="cri-o://6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81" gracePeriod=30 Oct 31 15:14:24.577937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.577904 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:24.584430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.584402 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm"] Oct 31 15:14:24.584644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.584624 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" containerID="cri-o://7674b61a0465765297e7292ba19b29888796d17a75da31ef5b693c505bb0fa7e" gracePeriod=30 Oct 31 15:14:24.586960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.586941 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:14:24.685486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.685445 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-plugins-config-volume\") pod \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " Oct 31 15:14:24.685660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.685581 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926c0093-7fa6-440e-afba-524657809b04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0093-7fa6-440e-afba-524657809b04-plugins-config-volume\") pod \"926c0093-7fa6-440e-afba-524657809b04\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " Oct 31 15:14:24.685660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.685619 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-kube-api-access-hlx9s\" (UniqueName: \"kubernetes.io/projected/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-kube-api-access-hlx9s\") pod \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\" (UID: \"e0b53a6e-dcc2-468d-9b71-afab5b65dccf\") " Oct 31 15:14:24.685782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.685766 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926c0093-7fa6-440e-afba-524657809b04-kube-api-access-crz4m\" (UniqueName: \"kubernetes.io/projected/926c0093-7fa6-440e-afba-524657809b04-kube-api-access-crz4m\") pod \"926c0093-7fa6-440e-afba-524657809b04\" (UID: \"926c0093-7fa6-440e-afba-524657809b04\") " Oct 31 15:14:24.685880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.685857 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-plugins-config-volume" pod "e0b53a6e-dcc2-468d-9b71-afab5b65dccf" (UID: "e0b53a6e-dcc2-468d-9b71-afab5b65dccf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:24.685938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.685921 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926c0093-7fa6-440e-afba-524657809b04-plugins-config-volume" pod "926c0093-7fa6-440e-afba-524657809b04" (UID: "926c0093-7fa6-440e-afba-524657809b04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:24.686372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.686289 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0093-7fa6-440e-afba-524657809b04-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:24.686372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.686313 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:24.688038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.688017 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-kube-api-access-hlx9s" pod "e0b53a6e-dcc2-468d-9b71-afab5b65dccf" (UID: "e0b53a6e-dcc2-468d-9b71-afab5b65dccf"). InnerVolumeSpecName "kube-api-access-hlx9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:24.688127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.688088 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926c0093-7fa6-440e-afba-524657809b04-kube-api-access-crz4m" pod "926c0093-7fa6-440e-afba-524657809b04" (UID: "926c0093-7fa6-440e-afba-524657809b04"). InnerVolumeSpecName "kube-api-access-crz4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:24.693050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.693027 4865 generic.go:356] "Generic (PLEG): container finished" podID="e0b53a6e-dcc2-468d-9b71-afab5b65dccf" containerID="da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02" exitCode=0 Oct 31 15:14:24.693130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.693073 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" Oct 31 15:14:24.693130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.693103 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" event={"ID":"e0b53a6e-dcc2-468d-9b71-afab5b65dccf","Type":"ContainerDied","Data":"da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02"} Oct 31 15:14:24.693208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.693154 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt" event={"ID":"e0b53a6e-dcc2-468d-9b71-afab5b65dccf","Type":"ContainerDied","Data":"9dc5635529505059e34a2830c2033f4b6e2f769bcd4bc99aae910e37d4618845"} Oct 31 15:14:24.693208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.693177 4865 scope.go:122] "RemoveContainer" containerID="da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02" Oct 31 15:14:24.700502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.694782 4865 generic.go:356] "Generic (PLEG): container finished" podID="926c0093-7fa6-440e-afba-524657809b04" containerID="941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4" exitCode=0 Oct 31 15:14:24.700502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.694810 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" event={"ID":"926c0093-7fa6-440e-afba-524657809b04","Type":"ContainerDied","Data":"941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4"} Oct 31 15:14:24.700502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.694832 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" Oct 31 15:14:24.700502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.694858 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f" event={"ID":"926c0093-7fa6-440e-afba-524657809b04","Type":"ContainerDied","Data":"6cfea3549b3c9825d7d90f12d9dbe39e89152db5728c875e8699a1e4c31c9121"} Oct 31 15:14:24.704378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.704355 4865 scope.go:122] "RemoveContainer" containerID="da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02" Oct 31 15:14:24.704690 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:24.704669 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02\": container with ID starting with da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02 not found: ID does not exist" containerID="da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02" Oct 31 15:14:24.704790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.704696 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02"} err="failed to get container status \"da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02\": rpc error: code = NotFound desc = could not find container \"da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02\": container with ID starting with da98b9c5e644189284d80b2c3fbc68039d31f38a308d0550df090e8bf830db02 not found: ID does not exist" Oct 31 15:14:24.704790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.704714 4865 scope.go:122] "RemoveContainer" containerID="941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4" Oct 31 15:14:24.714497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.714461 4865 scope.go:122] "RemoveContainer" containerID="941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4" Oct 31 15:14:24.714736 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:24.714719 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4\": container with ID starting with 941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4 not found: ID does not exist" containerID="941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4" Oct 31 15:14:24.714789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.714743 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4"} err="failed to get container status \"941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4\": rpc error: code = NotFound desc = could not find container \"941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4\": container with ID starting with 941e43f43315d5b0b58d491775e7a372587217c1fc97a9bed197ae4be0d878a4 not found: ID does not exist" Oct 31 15:14:24.804541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.797964 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlx9s\" (UniqueName: \"kubernetes.io/projected/e0b53a6e-dcc2-468d-9b71-afab5b65dccf-kube-api-access-hlx9s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:24.804541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:24.798006 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crz4m\" (UniqueName: \"kubernetes.io/projected/926c0093-7fa6-440e-afba-524657809b04-kube-api-access-crz4m\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:25.133316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.133233 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f"] Oct 31 15:14:25.136226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.136202 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-95df74d79-nn74f"] Oct 31 15:14:25.246726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.246697 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt"] Oct 31 15:14:25.250800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.250774 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-58c8564fcf-v9xnt"] Oct 31 15:14:25.258055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.258028 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:25.258186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.258095 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:25.329677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.329648 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:25.788207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.787551 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:25.876841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.876812 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8"] Oct 31 15:14:25.877422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877408 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e0b53a6e-dcc2-468d-9b71-afab5b65dccf" containerName="payload-processing" Oct 31 15:14:25.877502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877424 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b53a6e-dcc2-468d-9b71-afab5b65dccf" containerName="payload-processing" Oct 31 15:14:25.877502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877443 4865 cpu_manager.go:414] "containerMap: removing container" podUID="84c45d05-3186-46e6-993c-e3d87ac0b3d2" containerName="payload-pre-processing" Oct 31 15:14:25.877502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877448 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c45d05-3186-46e6-993c-e3d87ac0b3d2" containerName="payload-pre-processing" Oct 31 15:14:25.877502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877500 4865 cpu_manager.go:414] "containerMap: removing container" podUID="926c0093-7fa6-440e-afba-524657809b04" containerName="payload-pre-processing" Oct 31 15:14:25.877644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877509 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926c0093-7fa6-440e-afba-524657809b04" containerName="payload-pre-processing" Oct 31 15:14:25.877644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877524 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1825c2c2-c391-4ab5-b8ac-3e06344a342d" containerName="payload-processing" Oct 31 15:14:25.877644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877530 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1825c2c2-c391-4ab5-b8ac-3e06344a342d" containerName="payload-processing" Oct 31 15:14:25.877738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877680 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="926c0093-7fa6-440e-afba-524657809b04" containerName="payload-pre-processing" Oct 31 15:14:25.877738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877690 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1825c2c2-c391-4ab5-b8ac-3e06344a342d" containerName="payload-processing" Oct 31 15:14:25.877738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877699 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b53a6e-dcc2-468d-9b71-afab5b65dccf" containerName="payload-processing" Oct 31 15:14:25.877738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.877706 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c45d05-3186-46e6-993c-e3d87ac0b3d2" containerName="payload-pre-processing" Oct 31 15:14:25.883614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.883064 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925417 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-data\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925508 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-socket\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925584 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-token\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925716 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-podinfo\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925758 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5rj\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-kube-api-access-rm5rj\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925845 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-envoy\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925934 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-credential-socket\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.925988 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-certs\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.931491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.926043 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istiod-ca-cert\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:25.982386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.981765 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f76cce-openshift-default-dockercfg-67xtm\"" Oct 31 15:14:25.982582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.982455 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8"] Oct 31 15:14:25.983921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:25.983839 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.028440 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-token\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.028620 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-podinfo\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.028669 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5rj\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-kube-api-access-rm5rj\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.028760 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-envoy\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.028854 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-credential-socket\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.028914 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-certs\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.028973 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istiod-ca-cert\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.029023 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-data\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.029059 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-socket\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.029586 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-socket\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.033684 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-podinfo\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.033995 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-certs\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.034675 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istiod-ca-cert\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.037711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.034935 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-data\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.038752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.037969 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-credential-socket\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.062307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.062270 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-envoy\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.160747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.159148 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-token\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.261931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.261867 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5rj\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-kube-api-access-rm5rj\") pod \"e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.284490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.271791 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:26.288835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.288750 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:26.462982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.462940 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:26.503323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.503211 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926c0093-7fa6-440e-afba-524657809b04" path="/var/lib/kubelet/pods/926c0093-7fa6-440e-afba-524657809b04/volumes" Oct 31 15:14:26.503858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.503828 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b53a6e-dcc2-468d-9b71-afab5b65dccf" path="/var/lib/kubelet/pods/e0b53a6e-dcc2-468d-9b71-afab5b65dccf/volumes" Oct 31 15:14:26.868248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.868222 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8"] Oct 31 15:14:26.868507 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:14:26.868459 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85dc57d_13a8_4eec_b109_3ca2a2ad22f4.slice/crio-dc47f2d6e25095d14ecf11bd3289e35c449200bd1a042b4e311aaebdae8481fb WatchSource:0}: Error finding container dc47f2d6e25095d14ecf11bd3289e35c449200bd1a042b4e311aaebdae8481fb: Status 404 returned error can't find the container with id dc47f2d6e25095d14ecf11bd3289e35c449200bd1a042b4e311aaebdae8481fb Oct 31 15:14:26.871277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.871243 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:14:26.871360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.871307 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:14:26.871360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:26.871335 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:14:27.723500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:27.723448 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" event={"ID":"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4","Type":"ContainerStarted","Data":"406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d"} Oct 31 15:14:27.723892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:27.723526 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" event={"ID":"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4","Type":"ContainerStarted","Data":"dc47f2d6e25095d14ecf11bd3289e35c449200bd1a042b4e311aaebdae8481fb"} Oct 31 15:14:27.770260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:27.770229 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:27.770407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:27.770281 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:27.934807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:27.934756 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podStartSLOduration=2.934742438 podStartE2EDuration="2.934742438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:27.933352431 +0000 UTC m=+1713.552903842" watchObservedRunningTime="2026-10-31 15:14:27.934742438 +0000 UTC m=+1713.554293849" Oct 31 15:14:28.116658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:28.116570 4865 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:28.116658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:28.116636 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:28.290123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:28.290086 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:28.295316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:28.295291 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:28.733035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:28.733004 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:28.734623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:28.734600 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:14:29.018947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.018855 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:29.261863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.215990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:29.403579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.402836 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:29.450449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.450414 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8"] Oct 31 15:14:29.462291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.461875 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:29.506750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.506719 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8"] Oct 31 15:14:29.539210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.539180 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a538f1\"" Oct 31 15:14:29.541101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.541079 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a538f1-dockercfg-txckd\"" Oct 31 15:14:29.677592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.677500 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:29.756339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.678456 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4jv\" (UniqueName: \"kubernetes.io/projected/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-kube-api-access-rg4jv\") pod \"payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:29.784082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.730529 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh"] Oct 31 15:14:29.784082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.768257 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh"] Oct 31 15:14:29.784082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.768340 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:29.786954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.786918 4865 generic.go:356] "Generic (PLEG): container finished" podID="106bab7b-8945-48ee-96bb-0343910ef473" containerID="7674b61a0465765297e7292ba19b29888796d17a75da31ef5b693c505bb0fa7e" exitCode=0 Oct 31 15:14:29.787185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.787164 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" event={"ID":"106bab7b-8945-48ee-96bb-0343910ef473","Type":"ContainerDied","Data":"7674b61a0465765297e7292ba19b29888796d17a75da31ef5b693c505bb0fa7e"} Oct 31 15:14:29.793202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.792061 4865 generic.go:356] "Generic (PLEG): container finished" podID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerID="874302466e17ce8dc4c2f0216ea735f58f766d2538b41132c633d2aca2e74ba9" exitCode=0 Oct 31 15:14:29.793202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.792106 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" event={"ID":"aa520adc-b9e5-42a1-b1b2-c487a73fedd4","Type":"ContainerDied","Data":"874302466e17ce8dc4c2f0216ea735f58f766d2538b41132c633d2aca2e74ba9"} Oct 31 15:14:29.865397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.865350 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:29.865600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.865574 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6c8b\" (UniqueName: \"kubernetes.io/projected/e617b98e-2f12-4415-99fd-64f97d7f0ccc-kube-api-access-q6c8b\") pod \"payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:29.866345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.865710 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4jv\" (UniqueName: \"kubernetes.io/projected/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-kube-api-access-rg4jv\") pod \"payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:29.866345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.865762 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617b98e-2f12-4415-99fd-64f97d7f0ccc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:29.866345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.866279 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:29.924673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.924642 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4jv\" (UniqueName: \"kubernetes.io/projected/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-kube-api-access-rg4jv\") pod \"payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:29.965658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.965289 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:14:29.972435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.972402 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6c8b\" (UniqueName: \"kubernetes.io/projected/e617b98e-2f12-4415-99fd-64f97d7f0ccc-kube-api-access-q6c8b\") pod \"payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:29.972645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.972599 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617b98e-2f12-4415-99fd-64f97d7f0ccc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:29.973677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:29.973653 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617b98e-2f12-4415-99fd-64f97d7f0ccc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:30.008009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.007988 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:14:30.011719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.011700 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:14:30.035107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.035076 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6c8b\" (UniqueName: \"kubernetes.io/projected/e617b98e-2f12-4415-99fd-64f97d7f0ccc-kube-api-access-q6c8b\") pod \"payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:30.074246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074220 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-istio-token\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-istio-token\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074261 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-data\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074306 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/106bab7b-8945-48ee-96bb-0343910ef473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106bab7b-8945-48ee-96bb-0343910ef473-istio-podinfo\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074418 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106bab7b-8945-48ee-96bb-0343910ef473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106bab7b-8945-48ee-96bb-0343910ef473-istiod-ca-cert\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074529 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-envoy\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074542 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-data" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.074662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074570 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-kube-api-access-rl89f\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-kube-api-access-rl89f\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074668 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-credential-socket\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074716 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-socket\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074754 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-certs\") pod \"106bab7b-8945-48ee-96bb-0343910ef473\" (UID: \"106bab7b-8945-48ee-96bb-0343910ef473\") " Oct 31 15:14:30.074857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.074785 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106bab7b-8945-48ee-96bb-0343910ef473-istiod-ca-cert" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:30.075039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075009 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-credential-socket" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.075039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075020 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-socket" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.075229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075202 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-certs" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.075818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075785 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.075818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075817 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106bab7b-8945-48ee-96bb-0343910ef473-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.075992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075833 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.075992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075847 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.075992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.075860 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.076781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.076759 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/106bab7b-8945-48ee-96bb-0343910ef473-istio-podinfo" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:30.076901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.076854 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-envoy" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.076967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.076917 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-kube-api-access-rl89f" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "kube-api-access-rl89f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.077218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.077154 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-istio-token" pod "106bab7b-8945-48ee-96bb-0343910ef473" (UID: "106bab7b-8945-48ee-96bb-0343910ef473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.078318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.078299 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:30.087426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.087405 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:30.177154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177003 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-kube-api-access-bhzv4\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-kube-api-access-bhzv4\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.177154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177047 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-envoy\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.177154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177123 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-socket\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.177586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177314 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-podinfo\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.177586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177392 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-data\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.177586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177427 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-certs\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.177586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177490 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-credential-socket\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.177586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.177585 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istiod-ca-cert\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.178404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178050 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-socket\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.178404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178139 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-token\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-token\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.178404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178210 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-kube-api-access-j4f7b\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-kube-api-access-j4f7b\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.178404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178290 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istiod-ca-cert\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.178404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178365 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-certs\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.178825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178452 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-data\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.178825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178513 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-podinfo\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.178825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178562 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-credential-socket\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.178825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178610 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-envoy\") pod \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\" (UID: \"07fce4fd-989c-4828-b307-12bdd3c0cdd3\") " Oct 31 15:14:30.178825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.178641 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-token\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-token\") pod \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\" (UID: \"aa520adc-b9e5-42a1-b1b2-c487a73fedd4\") " Oct 31 15:14:30.179579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.179487 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106bab7b-8945-48ee-96bb-0343910ef473-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.179579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.179510 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106bab7b-8945-48ee-96bb-0343910ef473-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.179579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.179526 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl89f\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-kube-api-access-rl89f\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.179579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.179540 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106bab7b-8945-48ee-96bb-0343910ef473-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.188137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.188108 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-data" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.188457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.188301 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-certs" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.189002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.188802 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-credential-socket" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.189314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.189138 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-certs" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.189581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.189433 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-socket" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.189581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.189452 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-data" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.189581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.189528 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-podinfo" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:30.189852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.189832 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-credential-socket" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.190422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.190401 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-socket" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.190689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.190670 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istiod-ca-cert" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:30.192381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.192110 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-token" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.192704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.192675 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istiod-ca-cert" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:30.193106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.193073 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-envoy" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.194022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.193989 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-token" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.194022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.194003 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-envoy" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.194275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.194257 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-podinfo" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:30.195142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.195121 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-kube-api-access-j4f7b" pod "aa520adc-b9e5-42a1-b1b2-c487a73fedd4" (UID: "aa520adc-b9e5-42a1-b1b2-c487a73fedd4"). InnerVolumeSpecName "kube-api-access-j4f7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.196322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.196302 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-kube-api-access-bhzv4" pod "07fce4fd-989c-4828-b307-12bdd3c0cdd3" (UID: "07fce4fd-989c-4828-b307-12bdd3c0cdd3"). InnerVolumeSpecName "kube-api-access-bhzv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.281223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281186 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzv4\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-kube-api-access-bhzv4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281224 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281240 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281253 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281267 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281280 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281294 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281308 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281324 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281337 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281351 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4f7b\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-kube-api-access-j4f7b\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281365 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281380 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281395 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281408 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281422 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281436 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07fce4fd-989c-4828-b307-12bdd3c0cdd3-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.281812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.281450 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa520adc-b9e5-42a1-b1b2-c487a73fedd4-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.475953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.475918 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8"] Oct 31 15:14:30.477232 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:14:30.477199 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a4bdc2_b7bd_4a14_937c_96f5cfca8cc9.slice/crio-ea4786e161c5f39ea31744025db110af8c1fa1b22bc565dd33b2fcb2f3cda225 WatchSource:0}: Error finding container ea4786e161c5f39ea31744025db110af8c1fa1b22bc565dd33b2fcb2f3cda225: Status 404 returned error can't find the container with id ea4786e161c5f39ea31744025db110af8c1fa1b22bc565dd33b2fcb2f3cda225 Oct 31 15:14:30.525210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.525179 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh"] Oct 31 15:14:30.530107 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:14:30.530083 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode617b98e_2f12_4415_99fd_64f97d7f0ccc.slice/crio-78eb741992ab1235c05f972834623bfbfed4a0ce10bc77eed5bc5cf1f760339f WatchSource:0}: Error finding container 78eb741992ab1235c05f972834623bfbfed4a0ce10bc77eed5bc5cf1f760339f: Status 404 returned error can't find the container with id 78eb741992ab1235c05f972834623bfbfed4a0ce10bc77eed5bc5cf1f760339f Oct 31 15:14:30.802190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.802094 4865 generic.go:356] "Generic (PLEG): container finished" podID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerID="6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81" exitCode=0 Oct 31 15:14:30.802190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.802173 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" Oct 31 15:14:30.802707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.802193 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" event={"ID":"07fce4fd-989c-4828-b307-12bdd3c0cdd3","Type":"ContainerDied","Data":"6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81"} Oct 31 15:14:30.802707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.802248 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn" event={"ID":"07fce4fd-989c-4828-b307-12bdd3c0cdd3","Type":"ContainerDied","Data":"9366f5c6c8367bf1346512e3fc4c6b7e9c4ccc19b79116c461566a1996a6061d"} Oct 31 15:14:30.802707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.802277 4865 scope.go:122] "RemoveContainer" containerID="6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81" Oct 31 15:14:30.804178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.804157 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" event={"ID":"e617b98e-2f12-4415-99fd-64f97d7f0ccc","Type":"ContainerStarted","Data":"f81f154243e3f9318d6b014e6976351488d00a40f854dd60f2409aad0d96913c"} Oct 31 15:14:30.804292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.804194 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" event={"ID":"e617b98e-2f12-4415-99fd-64f97d7f0ccc","Type":"ContainerStarted","Data":"78eb741992ab1235c05f972834623bfbfed4a0ce10bc77eed5bc5cf1f760339f"} Oct 31 15:14:30.804292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.804216 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:30.805832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.805810 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" Oct 31 15:14:30.805944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.805829 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm" event={"ID":"106bab7b-8945-48ee-96bb-0343910ef473","Type":"ContainerDied","Data":"1172a582aac0c9a78b36f7964827734d71e9887d8c3c439243cf94469f8dad71"} Oct 31 15:14:30.807766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.807740 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" event={"ID":"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9","Type":"ContainerStarted","Data":"1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df"} Oct 31 15:14:30.807873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.807791 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" event={"ID":"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9","Type":"ContainerStarted","Data":"ea4786e161c5f39ea31744025db110af8c1fa1b22bc565dd33b2fcb2f3cda225"} Oct 31 15:14:30.807873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.807822 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:30.809262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.809197 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" Oct 31 15:14:30.809262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.809226 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4" event={"ID":"aa520adc-b9e5-42a1-b1b2-c487a73fedd4","Type":"ContainerDied","Data":"ecb1b9448fe8d5c0df0dd2ed8b0ec2aa26ca11fa170a7323fcfe0a835654d53e"} Oct 31 15:14:30.815695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.815674 4865 scope.go:122] "RemoveContainer" containerID="6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23" Oct 31 15:14:30.829642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.829523 4865 scope.go:122] "RemoveContainer" containerID="6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81" Oct 31 15:14:30.830230 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:30.830202 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81\": container with ID starting with 6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81 not found: ID does not exist" containerID="6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81" Oct 31 15:14:30.830341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.830244 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81"} err="failed to get container status \"6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81\": rpc error: code = NotFound desc = could not find container \"6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81\": container with ID starting with 6463b95dc47df4ba1828759a7f42052459735b9c8d5acd45cb231a2be2dc3c81 not found: ID does not exist" Oct 31 15:14:30.830341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.830270 4865 scope.go:122] "RemoveContainer" containerID="6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23" Oct 31 15:14:30.830627 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:30.830603 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23\": container with ID starting with 6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23 not found: ID does not exist" containerID="6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23" Oct 31 15:14:30.830749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.830718 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23"} err="failed to get container status \"6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23\": rpc error: code = NotFound desc = could not find container \"6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23\": container with ID starting with 6be13889a70908e2091d5ac272c464113e71708ccc2e1e1b3e21ca07eaafdc23 not found: ID does not exist" Oct 31 15:14:30.830749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.830748 4865 scope.go:122] "RemoveContainer" containerID="7674b61a0465765297e7292ba19b29888796d17a75da31ef5b693c505bb0fa7e" Oct 31 15:14:30.840099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.840083 4865 scope.go:122] "RemoveContainer" containerID="874302466e17ce8dc4c2f0216ea735f58f766d2538b41132c633d2aca2e74ba9" Oct 31 15:14:30.974560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:30.974512 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" podStartSLOduration=1.974497365 podStartE2EDuration="1.974497365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:30.970728912 +0000 UTC m=+1716.590280323" watchObservedRunningTime="2026-10-31 15:14:30.974497365 +0000 UTC m=+1716.594048826" Oct 31 15:14:31.096317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.095976 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" podStartSLOduration=2.095956074 podStartE2EDuration="2.095956074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:31.086970018 +0000 UTC m=+1716.706521431" watchObservedRunningTime="2026-10-31 15:14:31.095956074 +0000 UTC m=+1716.715507487" Oct 31 15:14:31.217664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.217598 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn"] Oct 31 15:14:31.224871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.224821 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-6q7cn"] Oct 31 15:14:31.326908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.326876 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4"] Oct 31 15:14:31.332176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.332150 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-clcd4"] Oct 31 15:14:31.443450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.443366 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm"] Oct 31 15:14:31.446662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.446636 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-vs4hm"] Oct 31 15:14:31.449631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.449579 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:31.625611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.625580 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:31.769528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:31.769457 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:32.149793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:32.149694 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:32.505567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:32.505522 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" path="/var/lib/kubelet/pods/07fce4fd-989c-4828-b307-12bdd3c0cdd3/volumes" Oct 31 15:14:32.506156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:32.506135 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106bab7b-8945-48ee-96bb-0343910ef473" path="/var/lib/kubelet/pods/106bab7b-8945-48ee-96bb-0343910ef473/volumes" Oct 31 15:14:32.506629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:32.506602 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" path="/var/lib/kubelet/pods/aa520adc-b9e5-42a1-b1b2-c487a73fedd4/volumes" Oct 31 15:14:34.544402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:34.544372 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:36.079617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.079578 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:36.277495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.276384 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:36.533801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.533769 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw"] Oct 31 15:14:36.534349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534334 4865 cpu_manager.go:414] "containerMap: removing container" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" Oct 31 15:14:36.534410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534351 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" Oct 31 15:14:36.534410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534395 4865 cpu_manager.go:414] "containerMap: removing container" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" Oct 31 15:14:36.534410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534400 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" Oct 31 15:14:36.534586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534425 4865 cpu_manager.go:414] "containerMap: removing container" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" Oct 31 15:14:36.534586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534430 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" Oct 31 15:14:36.534586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534527 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa520adc-b9e5-42a1-b1b2-c487a73fedd4" containerName="istio-proxy" Oct 31 15:14:36.534586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534539 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" Oct 31 15:14:36.534586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534550 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" Oct 31 15:14:36.534586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.534556 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="106bab7b-8945-48ee-96bb-0343910ef473" containerName="istio-proxy" Oct 31 15:14:36.539418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.539399 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:36.598455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.598414 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw"] Oct 31 15:14:36.682507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.676745 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f76cce-dockercfg-r99w6\"" Oct 31 15:14:36.682507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.676867 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f76cce\"" Oct 31 15:14:36.714494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.711721 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fsk\" (UniqueName: \"kubernetes.io/projected/6e31af48-0ef0-40dc-97c3-473e46736456-kube-api-access-k9fsk\") pod \"payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:36.714494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.711795 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e31af48-0ef0-40dc-97c3-473e46736456-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:36.839963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.838243 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fsk\" (UniqueName: \"kubernetes.io/projected/6e31af48-0ef0-40dc-97c3-473e46736456-kube-api-access-k9fsk\") pod \"payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:36.839963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.838314 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e31af48-0ef0-40dc-97c3-473e46736456-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:36.844559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.844529 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e31af48-0ef0-40dc-97c3-473e46736456-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:36.901265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:36.901229 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fsk\" (UniqueName: \"kubernetes.io/projected/6e31af48-0ef0-40dc-97c3-473e46736456-kube-api-access-k9fsk\") pod \"payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:37.152942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.152852 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:37.529602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.529573 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw"] Oct 31 15:14:37.529966 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:14:37.529940 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e31af48_0ef0_40dc_97c3_473e46736456.slice/crio-436ed4defcc550cdb20b0be235fb2d06b27861b6eb8eab60cf153bd44c12a102 WatchSource:0}: Error finding container 436ed4defcc550cdb20b0be235fb2d06b27861b6eb8eab60cf153bd44c12a102: Status 404 returned error can't find the container with id 436ed4defcc550cdb20b0be235fb2d06b27861b6eb8eab60cf153bd44c12a102 Oct 31 15:14:37.529966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.529960 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:37.660507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.660333 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:37.871302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.871219 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" event={"ID":"6e31af48-0ef0-40dc-97c3-473e46736456","Type":"ContainerStarted","Data":"04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6"} Oct 31 15:14:37.871302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.871276 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" event={"ID":"6e31af48-0ef0-40dc-97c3-473e46736456","Type":"ContainerStarted","Data":"436ed4defcc550cdb20b0be235fb2d06b27861b6eb8eab60cf153bd44c12a102"} Oct 31 15:14:37.871575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.871426 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:37.989976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:37.989924 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" podStartSLOduration=1.989910287 podStartE2EDuration="1.989910287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:37.98660763 +0000 UTC m=+1723.606159040" watchObservedRunningTime="2026-10-31 15:14:37.989910287 +0000 UTC m=+1723.609461692" Oct 31 15:14:38.795492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:38.782015 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:40.504287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:40.504258 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:41.823647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:41.823618 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:14:41.824073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:41.824021 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:14:45.197723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:45.197678 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:46.132343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:46.132297 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:46.273936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:46.273903 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:47.490636 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:47.490595 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:14:47.714294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:47.714233 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:48.233833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:48.233803 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:48.240741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:48.240718 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl"] Oct 31 15:14:48.240931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:48.240912 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" containerID="cri-o://396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5" gracePeriod=30 Oct 31 15:14:48.881217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:48.881186 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:14:49.009269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:49.009226 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:50.323592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:50.323561 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:51.067069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:51.067031 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:14:53.510526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.510448 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:14:53.564883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.564852 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-credential-socket\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.564892 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-envoy\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.564953 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-certs\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565017 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-data\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565135 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-socket\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565155 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-credential-socket" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:53.565354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565261 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-token\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-token\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565312 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-kube-api-access-cbvfp\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-kube-api-access-cbvfp\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565351 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-podinfo\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565393 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-data" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:53.565578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565406 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-certs" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:53.565578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565565 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-socket" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:53.565729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565575 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e639f49-cb86-4ab5-a18e-efe152020a2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e639f49-cb86-4ab5-a18e-efe152020a2a-istiod-ca-cert\") pod \"0e639f49-cb86-4ab5-a18e-efe152020a2a\" (UID: \"0e639f49-cb86-4ab5-a18e-efe152020a2a\") " Oct 31 15:14:53.565946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.565920 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e639f49-cb86-4ab5-a18e-efe152020a2a-istiod-ca-cert" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:53.566739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.566660 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e639f49-cb86-4ab5-a18e-efe152020a2a-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.566878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.566851 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.566959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.566880 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.566959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.566896 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.566959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.566910 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.567913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.567888 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-podinfo" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:53.568036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.567965 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-kube-api-access-cbvfp" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "kube-api-access-cbvfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:53.568103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.568090 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-envoy" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:53.568289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.568274 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-token" pod "0e639f49-cb86-4ab5-a18e-efe152020a2a" (UID: "0e639f49-cb86-4ab5-a18e-efe152020a2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:53.668313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.668279 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.668313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.668316 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvfp\" (UniqueName: \"kubernetes.io/projected/0e639f49-cb86-4ab5-a18e-efe152020a2a-kube-api-access-cbvfp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.668543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.668331 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.668543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.668345 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e639f49-cb86-4ab5-a18e-efe152020a2a-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:53.989344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.989309 4865 generic.go:356] "Generic (PLEG): container finished" podID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerID="396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5" exitCode=0 Oct 31 15:14:53.989559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.989362 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" Oct 31 15:14:53.989559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.989390 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" event={"ID":"0e639f49-cb86-4ab5-a18e-efe152020a2a","Type":"ContainerDied","Data":"396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5"} Oct 31 15:14:53.989559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.989448 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl" event={"ID":"0e639f49-cb86-4ab5-a18e-efe152020a2a","Type":"ContainerDied","Data":"012f037a1041bad6e2ab714614b996eccb0136f6d6ebf026b68783d59407e6c5"} Oct 31 15:14:53.989559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:53.989487 4865 scope.go:122] "RemoveContainer" containerID="396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5" Oct 31 15:14:54.000800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.000780 4865 scope.go:122] "RemoveContainer" containerID="9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba" Oct 31 15:14:54.011051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.011034 4865 scope.go:122] "RemoveContainer" containerID="396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5" Oct 31 15:14:54.011322 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:54.011299 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5\": container with ID starting with 396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5 not found: ID does not exist" containerID="396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5" Oct 31 15:14:54.011370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.011332 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5"} err="failed to get container status \"396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5\": rpc error: code = NotFound desc = could not find container \"396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5\": container with ID starting with 396dc0f40684e5fa3dc5912389836acde7a7a78aaab01b711bd7c760c18d4aa5 not found: ID does not exist" Oct 31 15:14:54.011370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.011352 4865 scope.go:122] "RemoveContainer" containerID="9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba" Oct 31 15:14:54.011615 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:14:54.011598 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba\": container with ID starting with 9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba not found: ID does not exist" containerID="9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba" Oct 31 15:14:54.011669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.011621 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba"} err="failed to get container status \"9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba\": rpc error: code = NotFound desc = could not find container \"9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba\": container with ID starting with 9896279adbe47ff878a8377ab9e52d146ec7e2bd9e4e397b0713fd5c1d4759ba not found: ID does not exist" Oct 31 15:14:54.162593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.162563 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl"] Oct 31 15:14:54.212531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.212501 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-aac61b-openshift-default-8644c949b5-x82sl"] Oct 31 15:14:54.434688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.434609 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:54.466828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.466784 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl"] Oct 31 15:14:54.498827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.498787 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" path="/var/lib/kubelet/pods/0e639f49-cb86-4ab5-a18e-efe152020a2a/volumes" Oct 31 15:14:54.585098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.581562 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl"] Oct 31 15:14:54.792235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.792196 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:54.922454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:54.922043 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:55.047066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:55.046965 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:55.448978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:55.448880 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:55.880496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:55.879590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:14:56.410413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:56.410383 4865 scope.go:122] "RemoveContainer" containerID="f465cb222ec02f6d4a2ac396cd546c7b88d7a007ac95f638147e0dcb55017876" Oct 31 15:14:56.420498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:56.420451 4865 scope.go:122] "RemoveContainer" containerID="7ae5f0db27c8df75e84b6baefd9084e7dfe578e1d6c13ffe06d76c5cc9d6dbfb" Oct 31 15:14:56.429720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:56.429700 4865 scope.go:122] "RemoveContainer" containerID="ef6842911ca75cdaf602d42c03e8719461acd1f1f25abe9f7b41de6bc9d6cf6d" Oct 31 15:14:59.467177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.467126 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" podUID="5c300b0e-e261-468a-9b8b-26bb624b8b28" containerName="payload-processing" containerID="cri-o://2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c" gracePeriod=25 Oct 31 15:14:59.582612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.582582 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" podUID="a1c5d656-4842-47be-bb60-d973c4a9487d" containerName="payload-pre-processing" containerID="cri-o://1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37" gracePeriod=25 Oct 31 15:14:59.736174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.736105 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:14:59.866546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.866516 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:14:59.872932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.872904 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c300b0e-e261-468a-9b8b-26bb624b8b28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c300b0e-e261-468a-9b8b-26bb624b8b28-plugins-config-volume\") pod \"5c300b0e-e261-468a-9b8b-26bb624b8b28\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " Oct 31 15:14:59.873203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.873183 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c300b0e-e261-468a-9b8b-26bb624b8b28-kube-api-access-2sltq\" (UniqueName: \"kubernetes.io/projected/5c300b0e-e261-468a-9b8b-26bb624b8b28-kube-api-access-2sltq\") pod \"5c300b0e-e261-468a-9b8b-26bb624b8b28\" (UID: \"5c300b0e-e261-468a-9b8b-26bb624b8b28\") " Oct 31 15:14:59.873941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.873525 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c300b0e-e261-468a-9b8b-26bb624b8b28-plugins-config-volume" pod "5c300b0e-e261-468a-9b8b-26bb624b8b28" (UID: "5c300b0e-e261-468a-9b8b-26bb624b8b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:59.873941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.873910 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c300b0e-e261-468a-9b8b-26bb624b8b28-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:59.875926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.875898 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c300b0e-e261-468a-9b8b-26bb624b8b28-kube-api-access-2sltq" pod "5c300b0e-e261-468a-9b8b-26bb624b8b28" (UID: "5c300b0e-e261-468a-9b8b-26bb624b8b28"). InnerVolumeSpecName "kube-api-access-2sltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:59.974664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.974566 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c5d656-4842-47be-bb60-d973c4a9487d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c5d656-4842-47be-bb60-d973c4a9487d-plugins-config-volume\") pod \"a1c5d656-4842-47be-bb60-d973c4a9487d\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " Oct 31 15:14:59.974664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.974611 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c5d656-4842-47be-bb60-d973c4a9487d-kube-api-access-2gcp4\" (UniqueName: \"kubernetes.io/projected/a1c5d656-4842-47be-bb60-d973c4a9487d-kube-api-access-2gcp4\") pod \"a1c5d656-4842-47be-bb60-d973c4a9487d\" (UID: \"a1c5d656-4842-47be-bb60-d973c4a9487d\") " Oct 31 15:14:59.975120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.975092 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c5d656-4842-47be-bb60-d973c4a9487d-plugins-config-volume" pod "a1c5d656-4842-47be-bb60-d973c4a9487d" (UID: "a1c5d656-4842-47be-bb60-d973c4a9487d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:59.975120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.975116 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sltq\" (UniqueName: \"kubernetes.io/projected/5c300b0e-e261-468a-9b8b-26bb624b8b28-kube-api-access-2sltq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:59.977293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:14:59.977265 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c5d656-4842-47be-bb60-d973c4a9487d-kube-api-access-2gcp4" pod "a1c5d656-4842-47be-bb60-d973c4a9487d" (UID: "a1c5d656-4842-47be-bb60-d973c4a9487d"). InnerVolumeSpecName "kube-api-access-2gcp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:00.063462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.063376 4865 generic.go:356] "Generic (PLEG): container finished" podID="a1c5d656-4842-47be-bb60-d973c4a9487d" containerID="1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37" exitCode=0 Oct 31 15:15:00.063655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.063509 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" Oct 31 15:15:00.063655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.063504 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" event={"ID":"a1c5d656-4842-47be-bb60-d973c4a9487d","Type":"ContainerDied","Data":"1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37"} Oct 31 15:15:00.063655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.063640 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl" event={"ID":"a1c5d656-4842-47be-bb60-d973c4a9487d","Type":"ContainerDied","Data":"ad5089f39dda42c9dec3c5878a82d1fd76a7cad6e0bdf7be21bba789576cc2be"} Oct 31 15:15:00.063802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.063658 4865 scope.go:122] "RemoveContainer" containerID="1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37" Oct 31 15:15:00.065141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.065112 4865 generic.go:356] "Generic (PLEG): container finished" podID="5c300b0e-e261-468a-9b8b-26bb624b8b28" containerID="2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c" exitCode=0 Oct 31 15:15:00.065274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.065163 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" Oct 31 15:15:00.065274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.065161 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" event={"ID":"5c300b0e-e261-468a-9b8b-26bb624b8b28","Type":"ContainerDied","Data":"2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c"} Oct 31 15:15:00.065359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.065285 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl" event={"ID":"5c300b0e-e261-468a-9b8b-26bb624b8b28","Type":"ContainerDied","Data":"0ebdedfb73efda1290a68f1a52d9fad602dac6357dc78c3f1288067e1430559f"} Oct 31 15:15:00.075388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.075365 4865 scope.go:122] "RemoveContainer" containerID="1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37" Oct 31 15:15:00.075790 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:00.075762 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37\": container with ID starting with 1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37 not found: ID does not exist" containerID="1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37" Oct 31 15:15:00.075920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.075801 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37"} err="failed to get container status \"1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37\": rpc error: code = NotFound desc = could not find container \"1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37\": container with ID starting with 1d70d2abfe6ca8cf503e056ee806a9484587611e873ee1b29a2852b73b02bf37 not found: ID does not exist" Oct 31 15:15:00.075920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.075824 4865 scope.go:122] "RemoveContainer" containerID="2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c" Oct 31 15:15:00.076432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.076412 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c5d656-4842-47be-bb60-d973c4a9487d-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:00.076580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.076436 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcp4\" (UniqueName: \"kubernetes.io/projected/a1c5d656-4842-47be-bb60-d973c4a9487d-kube-api-access-2gcp4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:00.086549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.086525 4865 scope.go:122] "RemoveContainer" containerID="2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c" Oct 31 15:15:00.086857 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:00.086837 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c\": container with ID starting with 2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c not found: ID does not exist" containerID="2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c" Oct 31 15:15:00.086956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.086868 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c"} err="failed to get container status \"2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c\": rpc error: code = NotFound desc = could not find container \"2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c\": container with ID starting with 2e8fb1a1073a53e57a60629925696de936e189c9956e17a6ff3afbac2a7dc70c not found: ID does not exist" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.170737 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n"] Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171750 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171771 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171815 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5c300b0e-e261-468a-9b8b-26bb624b8b28" containerName="payload-processing" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171822 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c300b0e-e261-468a-9b8b-26bb624b8b28" containerName="payload-processing" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171833 4865 cpu_manager.go:414] "containerMap: removing container" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171840 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fce4fd-989c-4828-b307-12bdd3c0cdd3" containerName="istio-proxy" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171850 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171857 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171870 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a1c5d656-4842-47be-bb60-d973c4a9487d" containerName="payload-pre-processing" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.171877 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c5d656-4842-47be-bb60-d973c4a9487d" containerName="payload-pre-processing" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.172014 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c300b0e-e261-468a-9b8b-26bb624b8b28" containerName="payload-processing" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.172023 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.172032 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c5d656-4842-47be-bb60-d973c4a9487d" containerName="payload-pre-processing" Oct 31 15:15:00.179572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.179284 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.182964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.182938 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk"] Oct 31 15:15:00.184091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.184053 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e639f49-cb86-4ab5-a18e-efe152020a2a" containerName="istio-proxy" Oct 31 15:15:00.187573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.187554 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb"] Oct 31 15:15:00.187697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.187668 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.191997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.191977 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z"] Oct 31 15:15:00.192181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.192112 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.196597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.196576 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942"] Oct 31 15:15:00.196701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.196687 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.203145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.203130 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr"] Oct 31 15:15:00.203254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.203238 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.206758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.206742 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g"] Oct 31 15:15:00.206847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.206837 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.210390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.210371 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.229323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.229304 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n"] Oct 31 15:15:00.232901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.232875 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:15:00.235762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.235738 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:15:00.235889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.235875 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:15:00.237494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.236790 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk"] Oct 31 15:15:00.248902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.248873 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl"] Oct 31 15:15:00.252821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.252793 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7251e-7db596df7-58mfl"] Oct 31 15:15:00.280515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.280040 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z"] Oct 31 15:15:00.281988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.281961 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942"] Oct 31 15:15:00.283690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.283661 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb0866-2a8d-402a-b164-05d7f93a235f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.283814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.283700 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72zz\" (UniqueName: \"kubernetes.io/projected/0773c62c-48d2-467c-b65c-6c7edd90973b-kube-api-access-s72zz\") pod \"maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.283814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.283728 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7s55\" (UniqueName: \"kubernetes.io/projected/d295b219-81d4-4fe7-9740-c0a3c1656cdd-kube-api-access-m7s55\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.283950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.283841 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkw8\" (UniqueName: \"kubernetes.io/projected/19025539-eb4a-4d9b-9432-901709917ead-kube-api-access-vwkw8\") pod \"maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.283950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.283862 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d295b219-81d4-4fe7-9740-c0a3c1656cdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.283950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.283919 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19025539-eb4a-4d9b-9432-901709917ead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.284114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.283979 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30647190-f0e5-4840-867f-60ad013f7a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.284114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284008 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gg2g\" (UniqueName: \"kubernetes.io/projected/d299bbc5-a623-4968-ac2c-75c403ed3a4f-kube-api-access-7gg2g\") pod \"maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.284114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284035 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlc4\" (UniqueName: \"kubernetes.io/projected/0deb0866-2a8d-402a-b164-05d7f93a235f-kube-api-access-jdlc4\") pod \"maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.284114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284063 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0773c62c-48d2-467c-b65c-6c7edd90973b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.284114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284104 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90c7310-a424-4cce-9cb2-e40f72fd8f10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.284369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284117 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q4kl\" (UniqueName: \"kubernetes.io/projected/30647190-f0e5-4840-867f-60ad013f7a55-kube-api-access-6q4kl\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.284369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284163 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjlhd\" (UniqueName: \"kubernetes.io/projected/c90c7310-a424-4cce-9cb2-e40f72fd8f10-kube-api-access-gjlhd\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.284369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284177 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299bbc5-a623-4968-ac2c-75c403ed3a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.284804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.284782 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g"] Oct 31 15:15:00.295146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.295120 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb"] Oct 31 15:15:00.332956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.332882 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr"] Oct 31 15:15:00.358615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.358583 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl"] Oct 31 15:15:00.361867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.361847 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7251e-7b74869899-8phtl"] Oct 31 15:15:00.385665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.385636 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjlhd\" (UniqueName: \"kubernetes.io/projected/c90c7310-a424-4cce-9cb2-e40f72fd8f10-kube-api-access-gjlhd\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.385790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.385667 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299bbc5-a623-4968-ac2c-75c403ed3a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.385790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.385754 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb0866-2a8d-402a-b164-05d7f93a235f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.385790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.385784 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72zz\" (UniqueName: \"kubernetes.io/projected/0773c62c-48d2-467c-b65c-6c7edd90973b-kube-api-access-s72zz\") pod \"maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.385961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.385903 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7s55\" (UniqueName: \"kubernetes.io/projected/d295b219-81d4-4fe7-9740-c0a3c1656cdd-kube-api-access-m7s55\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.386026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.385996 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkw8\" (UniqueName: \"kubernetes.io/projected/19025539-eb4a-4d9b-9432-901709917ead-kube-api-access-vwkw8\") pod \"maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.386090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386026 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d295b219-81d4-4fe7-9740-c0a3c1656cdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.386144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386097 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19025539-eb4a-4d9b-9432-901709917ead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.386144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386130 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30647190-f0e5-4840-867f-60ad013f7a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.386272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386167 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gg2g\" (UniqueName: \"kubernetes.io/projected/d299bbc5-a623-4968-ac2c-75c403ed3a4f-kube-api-access-7gg2g\") pod \"maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.386272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386218 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlc4\" (UniqueName: \"kubernetes.io/projected/0deb0866-2a8d-402a-b164-05d7f93a235f-kube-api-access-jdlc4\") pod \"maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.386397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386270 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0773c62c-48d2-467c-b65c-6c7edd90973b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.386397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386333 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90c7310-a424-4cce-9cb2-e40f72fd8f10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.386397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386358 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q4kl\" (UniqueName: \"kubernetes.io/projected/30647190-f0e5-4840-867f-60ad013f7a55-kube-api-access-6q4kl\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.386625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386602 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299bbc5-a623-4968-ac2c-75c403ed3a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.386788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386604 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb0866-2a8d-402a-b164-05d7f93a235f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.386985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386960 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30647190-f0e5-4840-867f-60ad013f7a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.387047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.386986 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19025539-eb4a-4d9b-9432-901709917ead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.387227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.387204 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d295b219-81d4-4fe7-9740-c0a3c1656cdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.387227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.387218 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0773c62c-48d2-467c-b65c-6c7edd90973b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.387528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.387506 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90c7310-a424-4cce-9cb2-e40f72fd8f10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.445380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.445353 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjlhd\" (UniqueName: \"kubernetes.io/projected/c90c7310-a424-4cce-9cb2-e40f72fd8f10-kube-api-access-gjlhd\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.445586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.445396 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72zz\" (UniqueName: \"kubernetes.io/projected/0773c62c-48d2-467c-b65c-6c7edd90973b-kube-api-access-s72zz\") pod \"maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.445586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.445427 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkw8\" (UniqueName: \"kubernetes.io/projected/19025539-eb4a-4d9b-9432-901709917ead-kube-api-access-vwkw8\") pod \"maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.445724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.445586 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7s55\" (UniqueName: \"kubernetes.io/projected/d295b219-81d4-4fe7-9740-c0a3c1656cdd-kube-api-access-m7s55\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.447070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.447025 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gg2g\" (UniqueName: \"kubernetes.io/projected/d299bbc5-a623-4968-ac2c-75c403ed3a4f-kube-api-access-7gg2g\") pod \"maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.447374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.447351 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q4kl\" (UniqueName: \"kubernetes.io/projected/30647190-f0e5-4840-867f-60ad013f7a55-kube-api-access-6q4kl\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.448546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.448522 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlc4\" (UniqueName: \"kubernetes.io/projected/0deb0866-2a8d-402a-b164-05d7f93a235f-kube-api-access-jdlc4\") pod \"maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.499020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.498991 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c300b0e-e261-468a-9b8b-26bb624b8b28" path="/var/lib/kubelet/pods/5c300b0e-e261-468a-9b8b-26bb624b8b28/volumes" Oct 31 15:15:00.499612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.499590 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c5d656-4842-47be-bb60-d973c4a9487d" path="/var/lib/kubelet/pods/a1c5d656-4842-47be-bb60-d973c4a9487d/volumes" Oct 31 15:15:00.499728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.499679 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:00.503524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.503504 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:00.508223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.508012 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:00.516081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.516062 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:00.524114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.524093 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:00.534295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.534272 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:00.586058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.584720 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:00.586058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.585251 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:00.722505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.689710 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:00.831791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.831667 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:00.963593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.963562 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:00.992331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:00.992303 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n"] Oct 31 15:15:01.076783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:01.076746 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" event={"ID":"30647190-f0e5-4840-867f-60ad013f7a55","Type":"ContainerStarted","Data":"725c71af85e0af9519276ad3240af2ea347591003099e63b9523790414a66f4e"} Oct 31 15:15:01.145864 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:01.145838 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90c7310_a424_4cce_9cb2_e40f72fd8f10.slice/crio-2a0156a8c5ee53b3c5b7b9676e66d55b8316c052e7db849e8dc1837ee70cb236 WatchSource:0}: Error finding container 2a0156a8c5ee53b3c5b7b9676e66d55b8316c052e7db849e8dc1837ee70cb236: Status 404 returned error can't find the container with id 2a0156a8c5ee53b3c5b7b9676e66d55b8316c052e7db849e8dc1837ee70cb236 Oct 31 15:15:01.146682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:01.146659 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb"] Oct 31 15:15:01.222935 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:01.222911 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd299bbc5_a623_4968_ac2c_75c403ed3a4f.slice/crio-7edeafff1c51407ccb23493c5a8519ef0072ad7197a49b0ae38c7fe6f0300552 WatchSource:0}: Error finding container 7edeafff1c51407ccb23493c5a8519ef0072ad7197a49b0ae38c7fe6f0300552: Status 404 returned error can't find the container with id 7edeafff1c51407ccb23493c5a8519ef0072ad7197a49b0ae38c7fe6f0300552 Oct 31 15:15:01.223057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:01.223038 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z"] Oct 31 15:15:01.236429 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:01.236408 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19025539_eb4a_4d9b_9432_901709917ead.slice/crio-6dea2801556d4f6e615c562874d0c79546d02dbf05f8e950925c2c77143c99b6 WatchSource:0}: Error finding container 6dea2801556d4f6e615c562874d0c79546d02dbf05f8e950925c2c77143c99b6: Status 404 returned error can't find the container with id 6dea2801556d4f6e615c562874d0c79546d02dbf05f8e950925c2c77143c99b6 Oct 31 15:15:01.237113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:01.237086 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g"] Oct 31 15:15:01.311612 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:01.311585 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0deb0866_2a8d_402a_b164_05d7f93a235f.slice/crio-0db4e020e8e106c4df3792ecf7b387a843c8ff55c9dce421f2928d130f9dd3da WatchSource:0}: Error finding container 0db4e020e8e106c4df3792ecf7b387a843c8ff55c9dce421f2928d130f9dd3da: Status 404 returned error can't find the container with id 0db4e020e8e106c4df3792ecf7b387a843c8ff55c9dce421f2928d130f9dd3da Oct 31 15:15:01.312270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:01.312247 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk"] Oct 31 15:15:01.329183 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:01.329156 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0773c62c_48d2_467c_b65c_6c7edd90973b.slice/crio-c15f687d019508431f55d04376fa56434e45b1c60d72b8be7d494bd879534904 WatchSource:0}: Error finding container c15f687d019508431f55d04376fa56434e45b1c60d72b8be7d494bd879534904: Status 404 returned error can't find the container with id c15f687d019508431f55d04376fa56434e45b1c60d72b8be7d494bd879534904 Oct 31 15:15:01.329409 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:01.329377 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942"] Oct 31 15:15:01.346324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:01.346305 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr"] Oct 31 15:15:01.346658 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:01.346637 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd295b219_81d4_4fe7_9740_c0a3c1656cdd.slice/crio-979909af2c2d05c363bd4ffde337719649ceb236d46a274398f18c317e44ddbb WatchSource:0}: Error finding container 979909af2c2d05c363bd4ffde337719649ceb236d46a274398f18c317e44ddbb: Status 404 returned error can't find the container with id 979909af2c2d05c363bd4ffde337719649ceb236d46a274398f18c317e44ddbb Oct 31 15:15:02.090326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.090240 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerStarted","Data":"d1375c19ac258fafea037a645317221a01fd7b454cfb65cdf17ddfa6fcb3d1e9"} Oct 31 15:15:02.090806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.090342 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerStarted","Data":"2a0156a8c5ee53b3c5b7b9676e66d55b8316c052e7db849e8dc1837ee70cb236"} Oct 31 15:15:02.091954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.091922 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerStarted","Data":"ab135f6fa0caf0b33617fb2ebb881d3888847d480aabfb6f1a1c8c2e5d2a91d0"} Oct 31 15:15:02.092079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.091968 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerStarted","Data":"c15f687d019508431f55d04376fa56434e45b1c60d72b8be7d494bd879534904"} Oct 31 15:15:02.093532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.093507 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" event={"ID":"d299bbc5-a623-4968-ac2c-75c403ed3a4f","Type":"ContainerStarted","Data":"c745a83dab8e36afd49aaf27c503749e53ba899452eb0548b28fcc0e55ec4b54"} Oct 31 15:15:02.093627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.093555 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" event={"ID":"d299bbc5-a623-4968-ac2c-75c403ed3a4f","Type":"ContainerStarted","Data":"7edeafff1c51407ccb23493c5a8519ef0072ad7197a49b0ae38c7fe6f0300552"} Oct 31 15:15:02.095148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.095125 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerStarted","Data":"720b10a89d896f608c5fdebaf8d3836af447fe61d51f2695f516a17278a44888"} Oct 31 15:15:02.095249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.095164 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerStarted","Data":"0db4e020e8e106c4df3792ecf7b387a843c8ff55c9dce421f2928d130f9dd3da"} Oct 31 15:15:02.098555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.098526 4865 generic.go:356] "Generic (PLEG): container finished" podID="30647190-f0e5-4840-867f-60ad013f7a55" containerID="5f96c9c8f95df01a04b65ee4402ac68047605fbc8bf4014b5f6480bf7f03f38e" exitCode=60 Oct 31 15:15:02.098670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.098608 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" event={"ID":"30647190-f0e5-4840-867f-60ad013f7a55","Type":"ContainerDied","Data":"5f96c9c8f95df01a04b65ee4402ac68047605fbc8bf4014b5f6480bf7f03f38e"} Oct 31 15:15:02.098806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.098790 4865 scope.go:122] "RemoveContainer" containerID="5f96c9c8f95df01a04b65ee4402ac68047605fbc8bf4014b5f6480bf7f03f38e" Oct 31 15:15:02.100869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.100846 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerStarted","Data":"0b9bf8e4bb8284716b57e84a6819921b717e7dbbdbebb28b07a6975ea5bb2e98"} Oct 31 15:15:02.100975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.100893 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerStarted","Data":"979909af2c2d05c363bd4ffde337719649ceb236d46a274398f18c317e44ddbb"} Oct 31 15:15:02.102636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.102613 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerStarted","Data":"8c3c7f81ae970359dc926ee9cd1311f64c386a9173296e5a13c0fc327fdb77cc"} Oct 31 15:15:02.102708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.102658 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerStarted","Data":"6dea2801556d4f6e615c562874d0c79546d02dbf05f8e950925c2c77143c99b6"} Oct 31 15:15:02.210822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.210777 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" podStartSLOduration=2.210762117 podStartE2EDuration="2.210762117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:02.208062811 +0000 UTC m=+1747.827614223" watchObservedRunningTime="2026-10-31 15:15:02.210762117 +0000 UTC m=+1747.830313527" Oct 31 15:15:02.326536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.326296 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" podStartSLOduration=2.326277369 podStartE2EDuration="2.326277369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:02.322081119 +0000 UTC m=+1747.941632529" watchObservedRunningTime="2026-10-31 15:15:02.326277369 +0000 UTC m=+1747.945828781" Oct 31 15:15:02.455946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.455876 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" podStartSLOduration=2.455855651 podStartE2EDuration="2.455855651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:02.447538162 +0000 UTC m=+1748.067089574" watchObservedRunningTime="2026-10-31 15:15:02.455855651 +0000 UTC m=+1748.075407065" Oct 31 15:15:02.676272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.676165 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" podStartSLOduration=2.676146825 podStartE2EDuration="2.676146825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:02.675171409 +0000 UTC m=+1748.294722821" watchObservedRunningTime="2026-10-31 15:15:02.676146825 +0000 UTC m=+1748.295698237" Oct 31 15:15:02.791801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.791751 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" podStartSLOduration=2.791735578 podStartE2EDuration="2.791735578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:02.788550036 +0000 UTC m=+1748.408101446" watchObservedRunningTime="2026-10-31 15:15:02.791735578 +0000 UTC m=+1748.411286990" Oct 31 15:15:02.906962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:02.906916 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" podStartSLOduration=2.9069014810000002 podStartE2EDuration="2.906901481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:02.904200159 +0000 UTC m=+1748.523751583" watchObservedRunningTime="2026-10-31 15:15:02.906901481 +0000 UTC m=+1748.526452892" Oct 31 15:15:03.113142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.113112 4865 generic.go:356] "Generic (PLEG): container finished" podID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerID="d1375c19ac258fafea037a645317221a01fd7b454cfb65cdf17ddfa6fcb3d1e9" exitCode=60 Oct 31 15:15:03.113567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.113204 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerDied","Data":"d1375c19ac258fafea037a645317221a01fd7b454cfb65cdf17ddfa6fcb3d1e9"} Oct 31 15:15:03.113567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.113506 4865 scope.go:122] "RemoveContainer" containerID="d1375c19ac258fafea037a645317221a01fd7b454cfb65cdf17ddfa6fcb3d1e9" Oct 31 15:15:03.114974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.114954 4865 generic.go:356] "Generic (PLEG): container finished" podID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerID="ab135f6fa0caf0b33617fb2ebb881d3888847d480aabfb6f1a1c8c2e5d2a91d0" exitCode=60 Oct 31 15:15:03.115078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.115022 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerDied","Data":"ab135f6fa0caf0b33617fb2ebb881d3888847d480aabfb6f1a1c8c2e5d2a91d0"} Oct 31 15:15:03.115340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.115283 4865 scope.go:122] "RemoveContainer" containerID="ab135f6fa0caf0b33617fb2ebb881d3888847d480aabfb6f1a1c8c2e5d2a91d0" Oct 31 15:15:03.116801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.116744 4865 generic.go:356] "Generic (PLEG): container finished" podID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerID="c745a83dab8e36afd49aaf27c503749e53ba899452eb0548b28fcc0e55ec4b54" exitCode=60 Oct 31 15:15:03.117834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.117074 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" event={"ID":"d299bbc5-a623-4968-ac2c-75c403ed3a4f","Type":"ContainerDied","Data":"c745a83dab8e36afd49aaf27c503749e53ba899452eb0548b28fcc0e55ec4b54"} Oct 31 15:15:03.117834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.117261 4865 scope.go:122] "RemoveContainer" containerID="c745a83dab8e36afd49aaf27c503749e53ba899452eb0548b28fcc0e55ec4b54" Oct 31 15:15:03.118949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.118930 4865 generic.go:356] "Generic (PLEG): container finished" podID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerID="720b10a89d896f608c5fdebaf8d3836af447fe61d51f2695f516a17278a44888" exitCode=60 Oct 31 15:15:03.119027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.119001 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerDied","Data":"720b10a89d896f608c5fdebaf8d3836af447fe61d51f2695f516a17278a44888"} Oct 31 15:15:03.119206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.119189 4865 scope.go:122] "RemoveContainer" containerID="720b10a89d896f608c5fdebaf8d3836af447fe61d51f2695f516a17278a44888" Oct 31 15:15:03.121189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.121167 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" event={"ID":"30647190-f0e5-4840-867f-60ad013f7a55","Type":"ContainerStarted","Data":"3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365"} Oct 31 15:15:03.121417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.121400 4865 scope.go:122] "RemoveContainer" containerID="3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365" Oct 31 15:15:03.121645 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:03.121624 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n_odh-ai-gateway-infra(30647190-f0e5-4840-867f-60ad013f7a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" podUID="30647190-f0e5-4840-867f-60ad013f7a55" Oct 31 15:15:03.123032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.123011 4865 generic.go:356] "Generic (PLEG): container finished" podID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerID="0b9bf8e4bb8284716b57e84a6819921b717e7dbbdbebb28b07a6975ea5bb2e98" exitCode=60 Oct 31 15:15:03.123148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.123088 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerDied","Data":"0b9bf8e4bb8284716b57e84a6819921b717e7dbbdbebb28b07a6975ea5bb2e98"} Oct 31 15:15:03.123326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.123308 4865 scope.go:122] "RemoveContainer" containerID="0b9bf8e4bb8284716b57e84a6819921b717e7dbbdbebb28b07a6975ea5bb2e98" Oct 31 15:15:03.124986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.124966 4865 generic.go:356] "Generic (PLEG): container finished" podID="19025539-eb4a-4d9b-9432-901709917ead" containerID="8c3c7f81ae970359dc926ee9cd1311f64c386a9173296e5a13c0fc327fdb77cc" exitCode=60 Oct 31 15:15:03.125100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.125015 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerDied","Data":"8c3c7f81ae970359dc926ee9cd1311f64c386a9173296e5a13c0fc327fdb77cc"} Oct 31 15:15:03.125315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:03.125243 4865 scope.go:122] "RemoveContainer" containerID="8c3c7f81ae970359dc926ee9cd1311f64c386a9173296e5a13c0fc327fdb77cc" Oct 31 15:15:04.133980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.133949 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerStarted","Data":"a735d7a75f74e4ba768ee215ba98f7cad6d1f6748c9c5af0799ff7d0205aaee8"} Oct 31 15:15:04.135891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.135866 4865 generic.go:356] "Generic (PLEG): container finished" podID="30647190-f0e5-4840-867f-60ad013f7a55" containerID="3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365" exitCode=60 Oct 31 15:15:04.136020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.135949 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" event={"ID":"30647190-f0e5-4840-867f-60ad013f7a55","Type":"ContainerDied","Data":"3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365"} Oct 31 15:15:04.136020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.136010 4865 scope.go:122] "RemoveContainer" containerID="5f96c9c8f95df01a04b65ee4402ac68047605fbc8bf4014b5f6480bf7f03f38e" Oct 31 15:15:04.136258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.136238 4865 scope.go:122] "RemoveContainer" containerID="3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365" Oct 31 15:15:04.136643 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:04.136622 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n_odh-ai-gateway-infra(30647190-f0e5-4840-867f-60ad013f7a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" podUID="30647190-f0e5-4840-867f-60ad013f7a55" Oct 31 15:15:04.137889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.137863 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerStarted","Data":"65f68808783dfcec0f1fa5841077709e71b11110efb64c9375c0e3fcdfd7b33e"} Oct 31 15:15:04.139774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.139750 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerStarted","Data":"719d8064b4e396b051bc665b42c9b0aec5c2088b596e30177601c848e13ca138"} Oct 31 15:15:04.141578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.141558 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerStarted","Data":"946560e8ca0bb174fd176eeb17d808bdbfce921eabb921a2692fc2dca23f6e93"} Oct 31 15:15:04.143600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.143578 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerStarted","Data":"f3f02483373795873ee21d77b82608bb9c3fde970bc47e28cec59255d166473f"} Oct 31 15:15:04.145619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:04.145587 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" event={"ID":"d299bbc5-a623-4968-ac2c-75c403ed3a4f","Type":"ContainerStarted","Data":"0a8d7ad1d93dbded09b81486a83ec3e6957bfdf5171369a0445ea132e1cfeedb"} Oct 31 15:15:05.154636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.154592 4865 generic.go:356] "Generic (PLEG): container finished" podID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerID="946560e8ca0bb174fd176eeb17d808bdbfce921eabb921a2692fc2dca23f6e93" exitCode=60 Oct 31 15:15:05.155151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.154677 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerDied","Data":"946560e8ca0bb174fd176eeb17d808bdbfce921eabb921a2692fc2dca23f6e93"} Oct 31 15:15:05.155151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.154755 4865 scope.go:122] "RemoveContainer" containerID="d1375c19ac258fafea037a645317221a01fd7b454cfb65cdf17ddfa6fcb3d1e9" Oct 31 15:15:05.155151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.154967 4865 scope.go:122] "RemoveContainer" containerID="946560e8ca0bb174fd176eeb17d808bdbfce921eabb921a2692fc2dca23f6e93" Oct 31 15:15:05.155334 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:05.155310 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb_odh-ai-gateway-infra(c90c7310-a424-4cce-9cb2-e40f72fd8f10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" Oct 31 15:15:05.157464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.157440 4865 generic.go:356] "Generic (PLEG): container finished" podID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerID="f3f02483373795873ee21d77b82608bb9c3fde970bc47e28cec59255d166473f" exitCode=60 Oct 31 15:15:05.157589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.157499 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerDied","Data":"f3f02483373795873ee21d77b82608bb9c3fde970bc47e28cec59255d166473f"} Oct 31 15:15:05.157818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.157804 4865 scope.go:122] "RemoveContainer" containerID="f3f02483373795873ee21d77b82608bb9c3fde970bc47e28cec59255d166473f" Oct 31 15:15:05.158110 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:05.158092 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942_odh-ai-gateway-infra(0773c62c-48d2-467c-b65c-6c7edd90973b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" Oct 31 15:15:05.159601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.159578 4865 generic.go:356] "Generic (PLEG): container finished" podID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerID="0a8d7ad1d93dbded09b81486a83ec3e6957bfdf5171369a0445ea132e1cfeedb" exitCode=60 Oct 31 15:15:05.159878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.159858 4865 scope.go:122] "RemoveContainer" containerID="0a8d7ad1d93dbded09b81486a83ec3e6957bfdf5171369a0445ea132e1cfeedb" Oct 31 15:15:05.159969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.159849 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" event={"ID":"d299bbc5-a623-4968-ac2c-75c403ed3a4f","Type":"ContainerDied","Data":"0a8d7ad1d93dbded09b81486a83ec3e6957bfdf5171369a0445ea132e1cfeedb"} Oct 31 15:15:05.160166 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:05.160141 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z_odh-ai-gateway-infra(d299bbc5-a623-4968-ac2c-75c403ed3a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" Oct 31 15:15:05.162170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.162138 4865 generic.go:356] "Generic (PLEG): container finished" podID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerID="a735d7a75f74e4ba768ee215ba98f7cad6d1f6748c9c5af0799ff7d0205aaee8" exitCode=60 Oct 31 15:15:05.162277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.162226 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerDied","Data":"a735d7a75f74e4ba768ee215ba98f7cad6d1f6748c9c5af0799ff7d0205aaee8"} Oct 31 15:15:05.162429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.162413 4865 scope.go:122] "RemoveContainer" containerID="a735d7a75f74e4ba768ee215ba98f7cad6d1f6748c9c5af0799ff7d0205aaee8" Oct 31 15:15:05.162745 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:05.162722 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk_odh-ai-gateway-infra(0deb0866-2a8d-402a-b164-05d7f93a235f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" Oct 31 15:15:05.164547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.164451 4865 scope.go:122] "RemoveContainer" containerID="3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365" Oct 31 15:15:05.164754 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:05.164733 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n_odh-ai-gateway-infra(30647190-f0e5-4840-867f-60ad013f7a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" podUID="30647190-f0e5-4840-867f-60ad013f7a55" Oct 31 15:15:05.166202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.166183 4865 generic.go:356] "Generic (PLEG): container finished" podID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerID="65f68808783dfcec0f1fa5841077709e71b11110efb64c9375c0e3fcdfd7b33e" exitCode=60 Oct 31 15:15:05.166313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.166244 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerDied","Data":"65f68808783dfcec0f1fa5841077709e71b11110efb64c9375c0e3fcdfd7b33e"} Oct 31 15:15:05.166456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.166442 4865 scope.go:122] "RemoveContainer" containerID="65f68808783dfcec0f1fa5841077709e71b11110efb64c9375c0e3fcdfd7b33e" Oct 31 15:15:05.166732 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:05.166708 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr_odh-ai-gateway-infra(d295b219-81d4-4fe7-9740-c0a3c1656cdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" Oct 31 15:15:05.168098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.168076 4865 generic.go:356] "Generic (PLEG): container finished" podID="19025539-eb4a-4d9b-9432-901709917ead" containerID="719d8064b4e396b051bc665b42c9b0aec5c2088b596e30177601c848e13ca138" exitCode=60 Oct 31 15:15:05.168175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.168141 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerDied","Data":"719d8064b4e396b051bc665b42c9b0aec5c2088b596e30177601c848e13ca138"} Oct 31 15:15:05.168322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.168309 4865 scope.go:122] "RemoveContainer" containerID="719d8064b4e396b051bc665b42c9b0aec5c2088b596e30177601c848e13ca138" Oct 31 15:15:05.168622 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:05.168605 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g_odh-ai-gateway-infra(19025539-eb4a-4d9b-9432-901709917ead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" podUID="19025539-eb4a-4d9b-9432-901709917ead" Oct 31 15:15:05.169561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.169544 4865 scope.go:122] "RemoveContainer" containerID="ab135f6fa0caf0b33617fb2ebb881d3888847d480aabfb6f1a1c8c2e5d2a91d0" Oct 31 15:15:05.182953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.182937 4865 scope.go:122] "RemoveContainer" containerID="c745a83dab8e36afd49aaf27c503749e53ba899452eb0548b28fcc0e55ec4b54" Oct 31 15:15:05.195565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.195542 4865 scope.go:122] "RemoveContainer" containerID="720b10a89d896f608c5fdebaf8d3836af447fe61d51f2695f516a17278a44888" Oct 31 15:15:05.208755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.208718 4865 scope.go:122] "RemoveContainer" containerID="0b9bf8e4bb8284716b57e84a6819921b717e7dbbdbebb28b07a6975ea5bb2e98" Oct 31 15:15:05.223856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.223825 4865 scope.go:122] "RemoveContainer" containerID="8c3c7f81ae970359dc926ee9cd1311f64c386a9173296e5a13c0fc327fdb77cc" Oct 31 15:15:05.819268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:05.819233 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:06.176991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:06.176917 4865 scope.go:122] "RemoveContainer" containerID="65f68808783dfcec0f1fa5841077709e71b11110efb64c9375c0e3fcdfd7b33e" Oct 31 15:15:06.177403 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:06.177235 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr_odh-ai-gateway-infra(d295b219-81d4-4fe7-9740-c0a3c1656cdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" Oct 31 15:15:06.178515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:06.178495 4865 scope.go:122] "RemoveContainer" containerID="719d8064b4e396b051bc665b42c9b0aec5c2088b596e30177601c848e13ca138" Oct 31 15:15:06.178799 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:06.178780 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g_odh-ai-gateway-infra(19025539-eb4a-4d9b-9432-901709917ead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" podUID="19025539-eb4a-4d9b-9432-901709917ead" Oct 31 15:15:06.180036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:06.180016 4865 scope.go:122] "RemoveContainer" containerID="946560e8ca0bb174fd176eeb17d808bdbfce921eabb921a2692fc2dca23f6e93" Oct 31 15:15:06.180272 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:06.180247 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb_odh-ai-gateway-infra(c90c7310-a424-4cce-9cb2-e40f72fd8f10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" Oct 31 15:15:06.181449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:06.181433 4865 scope.go:122] "RemoveContainer" containerID="f3f02483373795873ee21d77b82608bb9c3fde970bc47e28cec59255d166473f" Oct 31 15:15:06.181667 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:06.181645 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942_odh-ai-gateway-infra(0773c62c-48d2-467c-b65c-6c7edd90973b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" Oct 31 15:15:06.182981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:06.182961 4865 scope.go:122] "RemoveContainer" containerID="0a8d7ad1d93dbded09b81486a83ec3e6957bfdf5171369a0445ea132e1cfeedb" Oct 31 15:15:06.183165 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:06.183145 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z_odh-ai-gateway-infra(d299bbc5-a623-4968-ac2c-75c403ed3a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" Oct 31 15:15:06.184522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:06.184506 4865 scope.go:122] "RemoveContainer" containerID="a735d7a75f74e4ba768ee215ba98f7cad6d1f6748c9c5af0799ff7d0205aaee8" Oct 31 15:15:06.184697 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:06.184680 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk_odh-ai-gateway-infra(0deb0866-2a8d-402a-b164-05d7f93a235f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" Oct 31 15:15:09.438267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:09.438236 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:09.927943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:09.927912 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:10.178546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.174815 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv"] Oct 31 15:15:10.195015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.194981 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.246530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.245359 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv"] Oct 31 15:15:10.275490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.261418 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:10.292525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.288210 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-644a8a\"/\"default-dockercfg-pmbnm\"" Oct 31 15:15:10.292525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.288455 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-644a8a\"/\"ipp-route-644a8a-kserve-self-signed-certs\"" Oct 31 15:15:10.310245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.309941 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-644a8a\"/\"kube-root-ca.crt\"" Oct 31 15:15:10.310245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.310019 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-644a8a\"/\"openshift-service-ca.crt\"" Oct 31 15:15:10.340494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.336365 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-dshm\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.340494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.336417 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-model-cache\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.340494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.336456 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tmp-dir\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.340494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.336551 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b2gd\" (UniqueName: \"kubernetes.io/projected/0620fcf9-acc1-44b0-9329-963bb2b8f55d-kube-api-access-9b2gd\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.340494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.336691 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-home\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.340494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.336855 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tls-certs\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.452981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.452880 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-home\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.453456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.453104 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tls-certs\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.453456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.453167 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-dshm\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.453456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.453192 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-model-cache\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.453456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.453221 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tmp-dir\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.453456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.453269 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2gd\" (UniqueName: \"kubernetes.io/projected/0620fcf9-acc1-44b0-9329-963bb2b8f55d-kube-api-access-9b2gd\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.476199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.475826 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-home\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.477271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.477242 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-dshm\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.491767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.478847 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tls-certs\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.491767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.479049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tmp-dir\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.492235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.492212 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-model-cache\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.540250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.540216 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2gd\" (UniqueName: \"kubernetes.io/projected/0620fcf9-acc1-44b0-9329-963bb2b8f55d-kube-api-access-9b2gd\") pod \"ipp-route-644a8a-kserve-585f96bb65-jkxlv\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:10.598789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:10.598756 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:11.109066 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:11.109037 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0620fcf9_acc1_44b0_9329_963bb2b8f55d.slice/crio-8f0e680c22e229ff299a5c75335120e317f6744589a4593caa57f2bd5ba729da WatchSource:0}: Error finding container 8f0e680c22e229ff299a5c75335120e317f6744589a4593caa57f2bd5ba729da: Status 404 returned error can't find the container with id 8f0e680c22e229ff299a5c75335120e317f6744589a4593caa57f2bd5ba729da Oct 31 15:15:11.110515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.110488 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv"] Oct 31 15:15:11.269952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.269922 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" event={"ID":"0620fcf9-acc1-44b0-9329-963bb2b8f55d","Type":"ContainerStarted","Data":"a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe"} Oct 31 15:15:11.270135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.269962 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" event={"ID":"0620fcf9-acc1-44b0-9329-963bb2b8f55d","Type":"ContainerStarted","Data":"8f0e680c22e229ff299a5c75335120e317f6744589a4593caa57f2bd5ba729da"} Oct 31 15:15:11.395859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.395767 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" podStartSLOduration=1.3957523969999999 podStartE2EDuration="1.395752397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:11.392578904 +0000 UTC m=+1757.012130316" watchObservedRunningTime="2026-10-31 15:15:11.395752397 +0000 UTC m=+1757.015303808" Oct 31 15:15:11.599899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.599871 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:11.833332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.833303 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl"] Oct 31 15:15:11.837678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.837656 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:11.891427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.891399 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2196e4\"" Oct 31 15:15:11.891634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.891506 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2196e4-dockercfg-wpj7t\"" Oct 31 15:15:11.895303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.895274 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl"] Oct 31 15:15:11.993506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.989039 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b7708f-f497-4077-8a99-460a5ac8e5da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:11.993506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:11.989098 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvsg\" (UniqueName: \"kubernetes.io/projected/16b7708f-f497-4077-8a99-460a5ac8e5da-kube-api-access-sjvsg\") pod \"payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:12.090287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.090216 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b7708f-f497-4077-8a99-460a5ac8e5da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:12.090287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.090262 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvsg\" (UniqueName: \"kubernetes.io/projected/16b7708f-f497-4077-8a99-460a5ac8e5da-kube-api-access-sjvsg\") pod \"payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:12.090923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.090903 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b7708f-f497-4077-8a99-460a5ac8e5da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:12.150483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.150436 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvsg\" (UniqueName: \"kubernetes.io/projected/16b7708f-f497-4077-8a99-460a5ac8e5da-kube-api-access-sjvsg\") pod \"payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:12.157340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.154501 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:12.541335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.541309 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl"] Oct 31 15:15:12.541814 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:12.541792 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b7708f_f497_4077_8a99_460a5ac8e5da.slice/crio-c19d5b3daa5d295090672e517d071c6865164e4c20b64c2697cad18a63e95bb8 WatchSource:0}: Error finding container c19d5b3daa5d295090672e517d071c6865164e4c20b64c2697cad18a63e95bb8: Status 404 returned error can't find the container with id c19d5b3daa5d295090672e517d071c6865164e4c20b64c2697cad18a63e95bb8 Oct 31 15:15:12.684113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.684070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:12.884486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:12.884388 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:13.290016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:13.289982 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" event={"ID":"16b7708f-f497-4077-8a99-460a5ac8e5da","Type":"ContainerStarted","Data":"ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4"} Oct 31 15:15:13.290249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:13.290035 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" event={"ID":"16b7708f-f497-4077-8a99-460a5ac8e5da","Type":"ContainerStarted","Data":"c19d5b3daa5d295090672e517d071c6865164e4c20b64c2697cad18a63e95bb8"} Oct 31 15:15:13.290249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:13.290082 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:13.409414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:13.409363 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" podStartSLOduration=2.409347889 podStartE2EDuration="2.409347889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:13.406284829 +0000 UTC m=+1759.025836237" watchObservedRunningTime="2026-10-31 15:15:13.409347889 +0000 UTC m=+1759.028899319" Oct 31 15:15:15.515494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:15.510226 4865 scope.go:122] "RemoveContainer" containerID="3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365" Oct 31 15:15:15.860236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:15.857500 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:15:16.314847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:16.314814 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" event={"ID":"30647190-f0e5-4840-867f-60ad013f7a55","Type":"ContainerStarted","Data":"3659515b1671530270447dc16808a35f95fef17b40cdbca68091f535df4a41e8"} Oct 31 15:15:16.432356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:16.432302 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" podStartSLOduration=16.432288488 podStartE2EDuration="16.432288488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:16.430140715 +0000 UTC m=+1762.049692126" watchObservedRunningTime="2026-10-31 15:15:16.432288488 +0000 UTC m=+1762.051839896" Oct 31 15:15:17.323238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:17.323206 4865 generic.go:356] "Generic (PLEG): container finished" podID="30647190-f0e5-4840-867f-60ad013f7a55" containerID="3659515b1671530270447dc16808a35f95fef17b40cdbca68091f535df4a41e8" exitCode=60 Oct 31 15:15:17.323706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:17.323277 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" event={"ID":"30647190-f0e5-4840-867f-60ad013f7a55","Type":"ContainerDied","Data":"3659515b1671530270447dc16808a35f95fef17b40cdbca68091f535df4a41e8"} Oct 31 15:15:17.323706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:17.323348 4865 scope.go:122] "RemoveContainer" containerID="3c44e206016a779f9f5ae1c76bd8c715d174d98bc53f73bd2da8cabad78bf365" Oct 31 15:15:17.323867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:17.323755 4865 scope.go:122] "RemoveContainer" containerID="3659515b1671530270447dc16808a35f95fef17b40cdbca68091f535df4a41e8" Oct 31 15:15:17.324105 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:17.324083 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n_odh-ai-gateway-infra(30647190-f0e5-4840-867f-60ad013f7a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" podUID="30647190-f0e5-4840-867f-60ad013f7a55" Oct 31 15:15:17.450050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:17.450008 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n"] Oct 31 15:15:17.489059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:17.489029 4865 scope.go:122] "RemoveContainer" containerID="0a8d7ad1d93dbded09b81486a83ec3e6957bfdf5171369a0445ea132e1cfeedb" Oct 31 15:15:17.492404 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:17.492375 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:15:18.333974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.333944 4865 generic.go:356] "Generic (PLEG): container finished" podID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerID="1ed6d2c3971260bf0286547e7183451995739f613caa5a8ac1d1fb7c3d39e41b" exitCode=60 Oct 31 15:15:18.334535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.334014 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" event={"ID":"d299bbc5-a623-4968-ac2c-75c403ed3a4f","Type":"ContainerDied","Data":"1ed6d2c3971260bf0286547e7183451995739f613caa5a8ac1d1fb7c3d39e41b"} Oct 31 15:15:18.334535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.334081 4865 scope.go:122] "RemoveContainer" containerID="0a8d7ad1d93dbded09b81486a83ec3e6957bfdf5171369a0445ea132e1cfeedb" Oct 31 15:15:18.334535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.334514 4865 scope.go:122] "RemoveContainer" containerID="1ed6d2c3971260bf0286547e7183451995739f613caa5a8ac1d1fb7c3d39e41b" Oct 31 15:15:18.334902 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:18.334877 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z_odh-ai-gateway-infra(d299bbc5-a623-4968-ac2c-75c403ed3a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" Oct 31 15:15:18.488188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.488159 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:18.489855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.489833 4865 scope.go:122] "RemoveContainer" containerID="946560e8ca0bb174fd176eeb17d808bdbfce921eabb921a2692fc2dca23f6e93" Oct 31 15:15:18.490005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.489926 4865 scope.go:122] "RemoveContainer" containerID="719d8064b4e396b051bc665b42c9b0aec5c2088b596e30177601c848e13ca138" Oct 31 15:15:18.490861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.490382 4865 scope.go:122] "RemoveContainer" containerID="a735d7a75f74e4ba768ee215ba98f7cad6d1f6748c9c5af0799ff7d0205aaee8" Oct 31 15:15:18.491062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.490887 4865 scope.go:122] "RemoveContainer" containerID="f3f02483373795873ee21d77b82608bb9c3fde970bc47e28cec59255d166473f" Oct 31 15:15:18.491327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.491238 4865 scope.go:122] "RemoveContainer" containerID="65f68808783dfcec0f1fa5841077709e71b11110efb64c9375c0e3fcdfd7b33e" Oct 31 15:15:18.605225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.605203 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30647190-f0e5-4840-867f-60ad013f7a55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30647190-f0e5-4840-867f-60ad013f7a55-maas-api-service-ca\") pod \"30647190-f0e5-4840-867f-60ad013f7a55\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " Oct 31 15:15:18.605349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.605306 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30647190-f0e5-4840-867f-60ad013f7a55-kube-api-access-6q4kl\" (UniqueName: \"kubernetes.io/projected/30647190-f0e5-4840-867f-60ad013f7a55-kube-api-access-6q4kl\") pod \"30647190-f0e5-4840-867f-60ad013f7a55\" (UID: \"30647190-f0e5-4840-867f-60ad013f7a55\") " Oct 31 15:15:18.605599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.605566 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30647190-f0e5-4840-867f-60ad013f7a55-maas-api-service-ca" pod "30647190-f0e5-4840-867f-60ad013f7a55" (UID: "30647190-f0e5-4840-867f-60ad013f7a55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:18.606684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.606659 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30647190-f0e5-4840-867f-60ad013f7a55-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:18.607799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.607779 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30647190-f0e5-4840-867f-60ad013f7a55-kube-api-access-6q4kl" pod "30647190-f0e5-4840-867f-60ad013f7a55" (UID: "30647190-f0e5-4840-867f-60ad013f7a55"). InnerVolumeSpecName "kube-api-access-6q4kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:18.708028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:18.707987 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q4kl\" (UniqueName: \"kubernetes.io/projected/30647190-f0e5-4840-867f-60ad013f7a55-kube-api-access-6q4kl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:19.288755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.288726 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:19.391498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.386491 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerStarted","Data":"67237cc494dd752ef211784c33286bbbd1a160bf04291d0ab5aef889fef0b4a1"} Oct 31 15:15:19.403989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.400445 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerStarted","Data":"46316731709b72abc8be1b07845b350d8d84e609806e43da47cd91adb592db50"} Oct 31 15:15:19.418309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.418270 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerStarted","Data":"89b26b2f82145ee4ac88e26dd5f7bb4cc417903ddc5d3a1fbe880fdd9c3ac376"} Oct 31 15:15:19.422635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.422595 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" event={"ID":"30647190-f0e5-4840-867f-60ad013f7a55","Type":"ContainerDied","Data":"725c71af85e0af9519276ad3240af2ea347591003099e63b9523790414a66f4e"} Oct 31 15:15:19.422791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.422658 4865 scope.go:122] "RemoveContainer" containerID="3659515b1671530270447dc16808a35f95fef17b40cdbca68091f535df4a41e8" Oct 31 15:15:19.422791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.422778 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n" Oct 31 15:15:19.429756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.427459 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerStarted","Data":"7a3ca407b96dc9a56ee99a400085468e9aed5e0705adcde336efdf49ad5176d1"} Oct 31 15:15:19.441426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.431332 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerStarted","Data":"8b650d04e3690e22f679b4c2b96649830583c488b4bc7feeeeee24601c9fd69d"} Oct 31 15:15:19.573394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.573360 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z"] Oct 31 15:15:19.956274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:19.956248 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:20.086331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.086250 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n"] Oct 31 15:15:20.089628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.089603 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29890995-7fv4n"] Oct 31 15:15:20.130081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.130056 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d299bbc5-a623-4968-ac2c-75c403ed3a4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299bbc5-a623-4968-ac2c-75c403ed3a4f-maas-api-service-ca\") pod \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " Oct 31 15:15:20.130267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.130183 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d299bbc5-a623-4968-ac2c-75c403ed3a4f-kube-api-access-7gg2g\" (UniqueName: \"kubernetes.io/projected/d299bbc5-a623-4968-ac2c-75c403ed3a4f-kube-api-access-7gg2g\") pod \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\" (UID: \"d299bbc5-a623-4968-ac2c-75c403ed3a4f\") " Oct 31 15:15:20.130446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.130422 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d299bbc5-a623-4968-ac2c-75c403ed3a4f-maas-api-service-ca" pod "d299bbc5-a623-4968-ac2c-75c403ed3a4f" (UID: "d299bbc5-a623-4968-ac2c-75c403ed3a4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:20.130695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.130676 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299bbc5-a623-4968-ac2c-75c403ed3a4f-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:20.132446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.132424 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d299bbc5-a623-4968-ac2c-75c403ed3a4f-kube-api-access-7gg2g" pod "d299bbc5-a623-4968-ac2c-75c403ed3a4f" (UID: "d299bbc5-a623-4968-ac2c-75c403ed3a4f"). InnerVolumeSpecName "kube-api-access-7gg2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:20.232354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.232310 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gg2g\" (UniqueName: \"kubernetes.io/projected/d299bbc5-a623-4968-ac2c-75c403ed3a4f-kube-api-access-7gg2g\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:20.440697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.440606 4865 generic.go:356] "Generic (PLEG): container finished" podID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerID="89b26b2f82145ee4ac88e26dd5f7bb4cc417903ddc5d3a1fbe880fdd9c3ac376" exitCode=60 Oct 31 15:15:20.440697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.440676 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerDied","Data":"89b26b2f82145ee4ac88e26dd5f7bb4cc417903ddc5d3a1fbe880fdd9c3ac376"} Oct 31 15:15:20.441232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.440734 4865 scope.go:122] "RemoveContainer" containerID="a735d7a75f74e4ba768ee215ba98f7cad6d1f6748c9c5af0799ff7d0205aaee8" Oct 31 15:15:20.441232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.441119 4865 scope.go:122] "RemoveContainer" containerID="89b26b2f82145ee4ac88e26dd5f7bb4cc417903ddc5d3a1fbe880fdd9c3ac376" Oct 31 15:15:20.441500 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:20.441445 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk_odh-ai-gateway-infra(0deb0866-2a8d-402a-b164-05d7f93a235f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" Oct 31 15:15:20.443654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.443632 4865 generic.go:356] "Generic (PLEG): container finished" podID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerID="7a3ca407b96dc9a56ee99a400085468e9aed5e0705adcde336efdf49ad5176d1" exitCode=60 Oct 31 15:15:20.443783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.443692 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerDied","Data":"7a3ca407b96dc9a56ee99a400085468e9aed5e0705adcde336efdf49ad5176d1"} Oct 31 15:15:20.444011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.443996 4865 scope.go:122] "RemoveContainer" containerID="7a3ca407b96dc9a56ee99a400085468e9aed5e0705adcde336efdf49ad5176d1" Oct 31 15:15:20.444249 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:20.444231 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr_odh-ai-gateway-infra(d295b219-81d4-4fe7-9740-c0a3c1656cdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" Oct 31 15:15:20.445547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.445525 4865 generic.go:356] "Generic (PLEG): container finished" podID="19025539-eb4a-4d9b-9432-901709917ead" containerID="8b650d04e3690e22f679b4c2b96649830583c488b4bc7feeeeee24601c9fd69d" exitCode=60 Oct 31 15:15:20.445674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.445604 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerDied","Data":"8b650d04e3690e22f679b4c2b96649830583c488b4bc7feeeeee24601c9fd69d"} Oct 31 15:15:20.445944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.445925 4865 scope.go:122] "RemoveContainer" containerID="8b650d04e3690e22f679b4c2b96649830583c488b4bc7feeeeee24601c9fd69d" Oct 31 15:15:20.446184 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:20.446143 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g_odh-ai-gateway-infra(19025539-eb4a-4d9b-9432-901709917ead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" podUID="19025539-eb4a-4d9b-9432-901709917ead" Oct 31 15:15:20.447456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.447435 4865 generic.go:356] "Generic (PLEG): container finished" podID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerID="67237cc494dd752ef211784c33286bbbd1a160bf04291d0ab5aef889fef0b4a1" exitCode=60 Oct 31 15:15:20.447578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.447508 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerDied","Data":"67237cc494dd752ef211784c33286bbbd1a160bf04291d0ab5aef889fef0b4a1"} Oct 31 15:15:20.447791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.447774 4865 scope.go:122] "RemoveContainer" containerID="67237cc494dd752ef211784c33286bbbd1a160bf04291d0ab5aef889fef0b4a1" Oct 31 15:15:20.448039 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:20.448017 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb_odh-ai-gateway-infra(c90c7310-a424-4cce-9cb2-e40f72fd8f10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" Oct 31 15:15:20.449529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.449503 4865 generic.go:356] "Generic (PLEG): container finished" podID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerID="46316731709b72abc8be1b07845b350d8d84e609806e43da47cd91adb592db50" exitCode=60 Oct 31 15:15:20.449630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.449573 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerDied","Data":"46316731709b72abc8be1b07845b350d8d84e609806e43da47cd91adb592db50"} Oct 31 15:15:20.449891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.449873 4865 scope.go:122] "RemoveContainer" containerID="46316731709b72abc8be1b07845b350d8d84e609806e43da47cd91adb592db50" Oct 31 15:15:20.450153 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:20.450132 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942_odh-ai-gateway-infra(0773c62c-48d2-467c-b65c-6c7edd90973b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" Oct 31 15:15:20.451034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.451014 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" Oct 31 15:15:20.451115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.451014 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z" event={"ID":"d299bbc5-a623-4968-ac2c-75c403ed3a4f","Type":"ContainerDied","Data":"7edeafff1c51407ccb23493c5a8519ef0072ad7197a49b0ae38c7fe6f0300552"} Oct 31 15:15:20.454971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.454955 4865 scope.go:122] "RemoveContainer" containerID="65f68808783dfcec0f1fa5841077709e71b11110efb64c9375c0e3fcdfd7b33e" Oct 31 15:15:20.467774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.467753 4865 scope.go:122] "RemoveContainer" containerID="719d8064b4e396b051bc665b42c9b0aec5c2088b596e30177601c848e13ca138" Oct 31 15:15:20.480358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.480339 4865 scope.go:122] "RemoveContainer" containerID="946560e8ca0bb174fd176eeb17d808bdbfce921eabb921a2692fc2dca23f6e93" Oct 31 15:15:20.498497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.498450 4865 scope.go:122] "RemoveContainer" containerID="f3f02483373795873ee21d77b82608bb9c3fde970bc47e28cec59255d166473f" Oct 31 15:15:20.502996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.502974 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30647190-f0e5-4840-867f-60ad013f7a55" path="/var/lib/kubelet/pods/30647190-f0e5-4840-867f-60ad013f7a55/volumes" Oct 31 15:15:20.511789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.511770 4865 scope.go:122] "RemoveContainer" containerID="1ed6d2c3971260bf0286547e7183451995739f613caa5a8ac1d1fb7c3d39e41b" Oct 31 15:15:20.521491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.521452 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb"] Oct 31 15:15:20.599948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.599915 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:20.613316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.613290 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:20.634022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.633993 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942"] Oct 31 15:15:20.748734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.748703 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr"] Oct 31 15:15:20.863369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.863336 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk"] Oct 31 15:15:20.981768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:20.981738 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g"] Oct 31 15:15:21.015719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.015648 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z"] Oct 31 15:15:21.019110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.019091 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-94a889-29890995-w8n6z"] Oct 31 15:15:21.485739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.485686 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:15:21.672765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.672679 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:21.773570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.773343 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19025539-eb4a-4d9b-9432-901709917ead-kube-api-access-vwkw8\" (UniqueName: \"kubernetes.io/projected/19025539-eb4a-4d9b-9432-901709917ead-kube-api-access-vwkw8\") pod \"19025539-eb4a-4d9b-9432-901709917ead\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " Oct 31 15:15:21.773570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.773423 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19025539-eb4a-4d9b-9432-901709917ead-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19025539-eb4a-4d9b-9432-901709917ead-maas-api-service-ca\") pod \"19025539-eb4a-4d9b-9432-901709917ead\" (UID: \"19025539-eb4a-4d9b-9432-901709917ead\") " Oct 31 15:15:21.774010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.773774 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19025539-eb4a-4d9b-9432-901709917ead-maas-api-service-ca" pod "19025539-eb4a-4d9b-9432-901709917ead" (UID: "19025539-eb4a-4d9b-9432-901709917ead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:21.774119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.774037 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19025539-eb4a-4d9b-9432-901709917ead-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.775905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.775883 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19025539-eb4a-4d9b-9432-901709917ead-kube-api-access-vwkw8" pod "19025539-eb4a-4d9b-9432-901709917ead" (UID: "19025539-eb4a-4d9b-9432-901709917ead"). InnerVolumeSpecName "kube-api-access-vwkw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:21.787803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.787772 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:21.790620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.790601 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:21.794214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.794193 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:21.797935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.797917 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:21.874901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.874871 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90c7310-a424-4cce-9cb2-e40f72fd8f10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90c7310-a424-4cce-9cb2-e40f72fd8f10-maas-api-service-ca\") pod \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.874935 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0773c62c-48d2-467c-b65c-6c7edd90973b-kube-api-access-s72zz\" (UniqueName: \"kubernetes.io/projected/0773c62c-48d2-467c-b65c-6c7edd90973b-kube-api-access-s72zz\") pod \"0773c62c-48d2-467c-b65c-6c7edd90973b\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.874969 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0773c62c-48d2-467c-b65c-6c7edd90973b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0773c62c-48d2-467c-b65c-6c7edd90973b-maas-api-service-ca\") pod \"0773c62c-48d2-467c-b65c-6c7edd90973b\" (UID: \"0773c62c-48d2-467c-b65c-6c7edd90973b\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.875020 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0deb0866-2a8d-402a-b164-05d7f93a235f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb0866-2a8d-402a-b164-05d7f93a235f-maas-api-service-ca\") pod \"0deb0866-2a8d-402a-b164-05d7f93a235f\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.875058 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d295b219-81d4-4fe7-9740-c0a3c1656cdd-kube-api-access-m7s55\" (UniqueName: \"kubernetes.io/projected/d295b219-81d4-4fe7-9740-c0a3c1656cdd-kube-api-access-m7s55\") pod \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.875113 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90c7310-a424-4cce-9cb2-e40f72fd8f10-kube-api-access-gjlhd\" (UniqueName: \"kubernetes.io/projected/c90c7310-a424-4cce-9cb2-e40f72fd8f10-kube-api-access-gjlhd\") pod \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\" (UID: \"c90c7310-a424-4cce-9cb2-e40f72fd8f10\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.875208 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d295b219-81d4-4fe7-9740-c0a3c1656cdd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d295b219-81d4-4fe7-9740-c0a3c1656cdd-maas-api-service-ca\") pod \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\" (UID: \"d295b219-81d4-4fe7-9740-c0a3c1656cdd\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.875261 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0deb0866-2a8d-402a-b164-05d7f93a235f-kube-api-access-jdlc4\" (UniqueName: \"kubernetes.io/projected/0deb0866-2a8d-402a-b164-05d7f93a235f-kube-api-access-jdlc4\") pod \"0deb0866-2a8d-402a-b164-05d7f93a235f\" (UID: \"0deb0866-2a8d-402a-b164-05d7f93a235f\") " Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.875363 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90c7310-a424-4cce-9cb2-e40f72fd8f10-maas-api-service-ca" pod "c90c7310-a424-4cce-9cb2-e40f72fd8f10" (UID: "c90c7310-a424-4cce-9cb2-e40f72fd8f10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.876013 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d295b219-81d4-4fe7-9740-c0a3c1656cdd-maas-api-service-ca" pod "d295b219-81d4-4fe7-9740-c0a3c1656cdd" (UID: "d295b219-81d4-4fe7-9740-c0a3c1656cdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.876064 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0deb0866-2a8d-402a-b164-05d7f93a235f-maas-api-service-ca" pod "0deb0866-2a8d-402a-b164-05d7f93a235f" (UID: "0deb0866-2a8d-402a-b164-05d7f93a235f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.876094 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90c7310-a424-4cce-9cb2-e40f72fd8f10-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.876114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.876117 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwkw8\" (UniqueName: \"kubernetes.io/projected/19025539-eb4a-4d9b-9432-901709917ead-kube-api-access-vwkw8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.877087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.876418 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0773c62c-48d2-467c-b65c-6c7edd90973b-maas-api-service-ca" pod "0773c62c-48d2-467c-b65c-6c7edd90973b" (UID: "0773c62c-48d2-467c-b65c-6c7edd90973b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:21.877855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.877822 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0773c62c-48d2-467c-b65c-6c7edd90973b-kube-api-access-s72zz" pod "0773c62c-48d2-467c-b65c-6c7edd90973b" (UID: "0773c62c-48d2-467c-b65c-6c7edd90973b"). InnerVolumeSpecName "kube-api-access-s72zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:21.878260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.878243 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0deb0866-2a8d-402a-b164-05d7f93a235f-kube-api-access-jdlc4" pod "0deb0866-2a8d-402a-b164-05d7f93a235f" (UID: "0deb0866-2a8d-402a-b164-05d7f93a235f"). InnerVolumeSpecName "kube-api-access-jdlc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:21.878701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.878675 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d295b219-81d4-4fe7-9740-c0a3c1656cdd-kube-api-access-m7s55" pod "d295b219-81d4-4fe7-9740-c0a3c1656cdd" (UID: "d295b219-81d4-4fe7-9740-c0a3c1656cdd"). InnerVolumeSpecName "kube-api-access-m7s55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:21.879047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.879028 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90c7310-a424-4cce-9cb2-e40f72fd8f10-kube-api-access-gjlhd" pod "c90c7310-a424-4cce-9cb2-e40f72fd8f10" (UID: "c90c7310-a424-4cce-9cb2-e40f72fd8f10"). InnerVolumeSpecName "kube-api-access-gjlhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:21.977248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.977218 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s72zz\" (UniqueName: \"kubernetes.io/projected/0773c62c-48d2-467c-b65c-6c7edd90973b-kube-api-access-s72zz\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.977248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.977247 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0773c62c-48d2-467c-b65c-6c7edd90973b-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.977456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.977259 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb0866-2a8d-402a-b164-05d7f93a235f-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.977456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.977268 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7s55\" (UniqueName: \"kubernetes.io/projected/d295b219-81d4-4fe7-9740-c0a3c1656cdd-kube-api-access-m7s55\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.977456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.977276 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjlhd\" (UniqueName: \"kubernetes.io/projected/c90c7310-a424-4cce-9cb2-e40f72fd8f10-kube-api-access-gjlhd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.977456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.977286 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d295b219-81d4-4fe7-9740-c0a3c1656cdd-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:21.977456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:21.977295 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlc4\" (UniqueName: \"kubernetes.io/projected/0deb0866-2a8d-402a-b164-05d7f93a235f-kube-api-access-jdlc4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:22.477135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.477100 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" Oct 31 15:15:22.477307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.477101 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk" event={"ID":"0deb0866-2a8d-402a-b164-05d7f93a235f","Type":"ContainerDied","Data":"0db4e020e8e106c4df3792ecf7b387a843c8ff55c9dce421f2928d130f9dd3da"} Oct 31 15:15:22.477307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.477240 4865 scope.go:122] "RemoveContainer" containerID="89b26b2f82145ee4ac88e26dd5f7bb4cc417903ddc5d3a1fbe880fdd9c3ac376" Oct 31 15:15:22.478639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.478623 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" Oct 31 15:15:22.478736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.478617 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr" event={"ID":"d295b219-81d4-4fe7-9740-c0a3c1656cdd","Type":"ContainerDied","Data":"979909af2c2d05c363bd4ffde337719649ceb236d46a274398f18c317e44ddbb"} Oct 31 15:15:22.480224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.480194 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" Oct 31 15:15:22.480350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.480239 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g" event={"ID":"19025539-eb4a-4d9b-9432-901709917ead","Type":"ContainerDied","Data":"6dea2801556d4f6e615c562874d0c79546d02dbf05f8e950925c2c77143c99b6"} Oct 31 15:15:22.481943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.481920 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" Oct 31 15:15:22.482076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.481945 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb" event={"ID":"c90c7310-a424-4cce-9cb2-e40f72fd8f10","Type":"ContainerDied","Data":"2a0156a8c5ee53b3c5b7b9676e66d55b8316c052e7db849e8dc1837ee70cb236"} Oct 31 15:15:22.490740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.490721 4865 scope.go:122] "RemoveContainer" containerID="7a3ca407b96dc9a56ee99a400085468e9aed5e0705adcde336efdf49ad5176d1" Oct 31 15:15:22.491204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.490815 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" Oct 31 15:15:22.507354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.507333 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" path="/var/lib/kubelet/pods/d299bbc5-a623-4968-ac2c-75c403ed3a4f/volumes" Oct 31 15:15:22.508421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.508365 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942" event={"ID":"0773c62c-48d2-467c-b65c-6c7edd90973b","Type":"ContainerDied","Data":"c15f687d019508431f55d04376fa56434e45b1c60d72b8be7d494bd879534904"} Oct 31 15:15:22.508776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.508756 4865 scope.go:122] "RemoveContainer" containerID="8b650d04e3690e22f679b4c2b96649830583c488b4bc7feeeeee24601c9fd69d" Oct 31 15:15:22.524807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.524789 4865 scope.go:122] "RemoveContainer" containerID="67237cc494dd752ef211784c33286bbbd1a160bf04291d0ab5aef889fef0b4a1" Oct 31 15:15:22.533950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.533933 4865 scope.go:122] "RemoveContainer" containerID="46316731709b72abc8be1b07845b350d8d84e609806e43da47cd91adb592db50" Oct 31 15:15:22.840423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.840352 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr"] Oct 31 15:15:22.843494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.843460 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890995-vwvpr"] Oct 31 15:15:22.954230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.954195 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk"] Oct 31 15:15:22.957307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:22.957287 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2196e4-29890995-srrdk"] Oct 31 15:15:23.067712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:23.067674 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb"] Oct 31 15:15:23.072264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:23.072236 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29890995-cgjjb"] Oct 31 15:15:23.189009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:23.188920 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g"] Oct 31 15:15:23.193654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:23.193614 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f7251e-29890995-4r68g"] Oct 31 15:15:23.296144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:23.296103 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942"] Oct 31 15:15:23.299616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:23.299589 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-644a8a-29890995-4b942"] Oct 31 15:15:24.028678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.028643 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:24.202122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.197525 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:24.302994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.302905 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:15:24.457689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.457658 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:24.500694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.500667 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" path="/var/lib/kubelet/pods/0773c62c-48d2-467c-b65c-6c7edd90973b/volumes" Oct 31 15:15:24.501067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.501053 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" path="/var/lib/kubelet/pods/0deb0866-2a8d-402a-b164-05d7f93a235f/volumes" Oct 31 15:15:24.501427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.501414 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19025539-eb4a-4d9b-9432-901709917ead" path="/var/lib/kubelet/pods/19025539-eb4a-4d9b-9432-901709917ead/volumes" Oct 31 15:15:24.502079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.502064 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" path="/var/lib/kubelet/pods/c90c7310-a424-4cce-9cb2-e40f72fd8f10/volumes" Oct 31 15:15:24.502459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:24.502442 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" path="/var/lib/kubelet/pods/d295b219-81d4-4fe7-9740-c0a3c1656cdd/volumes" Oct 31 15:15:29.046055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:29.046025 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt"] Oct 31 15:15:29.096591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:29.096558 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4601702-7kgjt"] Oct 31 15:15:29.155867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:29.155811 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:30.497272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:30.497241 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b696ce-c1b7-47ed-831e-3515a8867261" path="/var/lib/kubelet/pods/b9b696ce-c1b7-47ed-831e-3515a8867261/volumes" Oct 31 15:15:32.658198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:32.658159 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:33.157406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:33.157357 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:34.277271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277239 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg"] Oct 31 15:15:34.277904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277887 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.277904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277905 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277916 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277921 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277937 4865 cpu_manager.go:414] "containerMap: removing container" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277943 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277950 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277955 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277964 4865 cpu_manager.go:414] "containerMap: removing container" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277969 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277975 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277980 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277987 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277992 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.277998 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278003 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278012 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278017 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278022 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278026 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278032 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278037 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278045 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278049 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278061 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278065 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278071 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278075 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278085 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278089 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278097 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278103 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278110 4865 cpu_manager.go:414] "containerMap: removing container" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278114 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278124 4865 cpu_manager.go:414] "containerMap: removing container" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278128 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278218 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278227 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278233 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278238 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278243 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278248 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278253 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278258 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278262 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d295b219-81d4-4fe7-9740-c0a3c1656cdd" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278268 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278273 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278278 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278283 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278288 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278293 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:34.278816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.278299 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:34.283715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.283697 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.326529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.326458 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg"] Oct 31 15:15:34.373854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.373826 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-94a889\"" Oct 31 15:15:34.374017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.373882 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-94a889-dockercfg-hjmn2\"" Oct 31 15:15:34.413126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.413098 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qcg\" (UniqueName: \"kubernetes.io/projected/81a79b66-503f-454d-b279-4be18a34ad2d-kube-api-access-26qcg\") pod \"payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.413278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.413255 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a79b66-503f-454d-b279-4be18a34ad2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.514433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.514405 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qcg\" (UniqueName: \"kubernetes.io/projected/81a79b66-503f-454d-b279-4be18a34ad2d-kube-api-access-26qcg\") pod \"payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.514623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.514519 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a79b66-503f-454d-b279-4be18a34ad2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.515169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.515149 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a79b66-503f-454d-b279-4be18a34ad2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.573575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.573508 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qcg\" (UniqueName: \"kubernetes.io/projected/81a79b66-503f-454d-b279-4be18a34ad2d-kube-api-access-26qcg\") pod \"payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.598013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.597989 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:34.979093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:34.979064 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg"] Oct 31 15:15:34.979957 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:15:34.979927 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a79b66_503f_454d_b279_4be18a34ad2d.slice/crio-eb74894b30f61b3cf3bd910a515ed4620d3218a67acbd4ef91fa8f74edb6d8c5 WatchSource:0}: Error finding container eb74894b30f61b3cf3bd910a515ed4620d3218a67acbd4ef91fa8f74edb6d8c5: Status 404 returned error can't find the container with id eb74894b30f61b3cf3bd910a515ed4620d3218a67acbd4ef91fa8f74edb6d8c5 Oct 31 15:15:35.075390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.075357 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq"] Oct 31 15:15:35.076436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076414 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:35.076544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076437 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90c7310-a424-4cce-9cb2-e40f72fd8f10" containerName="cleanup" Oct 31 15:15:35.076589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076578 4865 cpu_manager.go:414] "containerMap: removing container" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:35.076589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076586 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:35.076666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076601 4865 cpu_manager.go:414] "containerMap: removing container" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:35.076666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076609 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:35.076776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076766 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="19025539-eb4a-4d9b-9432-901709917ead" containerName="cleanup" Oct 31 15:15:35.076818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076780 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0773c62c-48d2-467c-b65c-6c7edd90973b" containerName="cleanup" Oct 31 15:15:35.076818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076789 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0deb0866-2a8d-402a-b164-05d7f93a235f" containerName="cleanup" Oct 31 15:15:35.076818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076799 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d299bbc5-a623-4968-ac2c-75c403ed3a4f" containerName="cleanup" Oct 31 15:15:35.076818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.076808 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="30647190-f0e5-4840-867f-60ad013f7a55" containerName="cleanup" Oct 31 15:15:35.080258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.080242 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.137065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.136936 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq"] Oct 31 15:15:35.223363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.223328 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q24g\" (UniqueName: \"kubernetes.io/projected/6230a2fd-0f3d-486c-b501-7228db16112e-kube-api-access-4q24g\") pod \"payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.223608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.223522 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6230a2fd-0f3d-486c-b501-7228db16112e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.324614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.324527 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q24g\" (UniqueName: \"kubernetes.io/projected/6230a2fd-0f3d-486c-b501-7228db16112e-kube-api-access-4q24g\") pod \"payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.325090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.324697 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6230a2fd-0f3d-486c-b501-7228db16112e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.325426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.325401 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6230a2fd-0f3d-486c-b501-7228db16112e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.383019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.382991 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q24g\" (UniqueName: \"kubernetes.io/projected/6230a2fd-0f3d-486c-b501-7228db16112e-kube-api-access-4q24g\") pod \"payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.394905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.394884 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:35.602226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.602126 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" event={"ID":"81a79b66-503f-454d-b279-4be18a34ad2d","Type":"ContainerStarted","Data":"6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05"} Oct 31 15:15:35.602226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.602190 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" event={"ID":"81a79b66-503f-454d-b279-4be18a34ad2d","Type":"ContainerStarted","Data":"eb74894b30f61b3cf3bd910a515ed4620d3218a67acbd4ef91fa8f74edb6d8c5"} Oct 31 15:15:35.602226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.602220 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:35.656908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.656842 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:35.755680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.755619 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" podStartSLOduration=1.755601391 podStartE2EDuration="1.755601391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:35.750828435 +0000 UTC m=+1781.370379864" watchObservedRunningTime="2026-10-31 15:15:35.755601391 +0000 UTC m=+1781.375152802" Oct 31 15:15:35.788117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.786740 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq"] Oct 31 15:15:35.816321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:35.816295 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:36.018987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:36.018958 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:36.612337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:36.612296 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" event={"ID":"6230a2fd-0f3d-486c-b501-7228db16112e","Type":"ContainerStarted","Data":"690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b"} Oct 31 15:15:36.612837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:36.612361 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" event={"ID":"6230a2fd-0f3d-486c-b501-7228db16112e","Type":"ContainerStarted","Data":"fc9c2caf166944827093e16a2250ebaad5d0bc90f12304fb9e6ccad8646f682a"} Oct 31 15:15:36.612837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:36.612403 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:36.730746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:36.730691 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" podStartSLOduration=1.73067554 podStartE2EDuration="1.73067554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:36.72880107 +0000 UTC m=+1782.348352488" watchObservedRunningTime="2026-10-31 15:15:36.73067554 +0000 UTC m=+1782.350226951" Oct 31 15:15:40.591740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:40.591706 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:41.347829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:41.347797 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:15:45.415494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:45.414982 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:46.618845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:46.618816 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:15:47.491722 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:15:47.491678 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:15:47.622925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:47.622842 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:15:50.106006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:50.105963 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:51.670584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.670548 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 15:15:51.671106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.670611 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 15:15:51.826088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.826055 4865 generic.go:356] "Generic (PLEG): container finished" podID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerID="0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681" exitCode=0 Oct 31 15:15:51.826267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.826134 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" event={"ID":"9a588319-ac1c-4ff5-af1a-e7902e68b263","Type":"ContainerDied","Data":"0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681"} Oct 31 15:15:51.826648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.826632 4865 scope.go:122] "RemoveContainer" containerID="0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681" Oct 31 15:15:51.828181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.828153 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:15:51.828256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.828223 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:15:51.828296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:51.828263 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:15:52.836462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:52.836423 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" event={"ID":"9a588319-ac1c-4ff5-af1a-e7902e68b263","Type":"ContainerStarted","Data":"9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da"} Oct 31 15:15:53.372291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:53.372255 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:15:53.372510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:53.372429 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 15:15:53.372510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:53.372484 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 15:15:54.215496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.193671 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:54.378511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.376878 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 15:15:54.378511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.376943 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 15:15:54.438599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.438548 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:54.827879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.827849 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:15:54.835279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.835254 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:15:54.838909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.838888 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:15:54.845715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.845692 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:15:54.867413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:54.867384 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:55.372264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:55.372232 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 15:15:55.372692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:55.372296 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 15:15:55.919276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:55.919246 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:56.266628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:56.266596 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:56.372507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:56.372451 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 15:15:56.372920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:56.372536 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 15:15:56.538877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:56.538788 4865 scope.go:122] "RemoveContainer" containerID="8a62c8bf33af588ba75491521f594d75b960edc79af31443a4aab8a9e596027e" Oct 31 15:15:56.549301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:56.549280 4865 scope.go:122] "RemoveContainer" containerID="44f8a09b9f0e8077bdf71261a036ad42c10ba1e689e3e48acf79b2156196abd3" Oct 31 15:15:57.371674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:57.371627 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 15:15:57.371883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:57.371694 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 15:15:58.377084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:58.377053 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:15:58.377576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:58.377498 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:15:58.378199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:58.378173 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:15:59.092677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:59.092644 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:59.228987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:59.228957 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:15:59.533189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:15:59.533156 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:00.654840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:00.654808 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:00.827272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:00.827237 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:01.318061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:01.318025 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:01.341601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:01.341566 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl"] Oct 31 15:16:01.341790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:01.341768 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" containerID="cri-o://108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c" gracePeriod=30 Oct 31 15:16:03.270343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:03.270296 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:03.584308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:03.584237 4865 patch_prober.go:29] interesting pod/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:16:03.584308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:03.584293 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:16:04.028203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:04.028165 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:04.380496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:04.380391 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:05.950218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:05.950174 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:06.618325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.618264 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:16:06.646616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.646579 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-credential-socket\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.646890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.646869 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-envoy\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.647048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.647034 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00341e7-e710-4156-8c80-6947b6d581e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00341e7-e710-4156-8c80-6947b6d581e7-istiod-ca-cert\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.647189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.647175 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-istio-token\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-istio-token\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.647344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.647331 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-data\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.647489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.647454 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-socket\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.647691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.647675 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a00341e7-e710-4156-8c80-6947b6d581e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00341e7-e710-4156-8c80-6947b6d581e7-istio-podinfo\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.647840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.647827 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-kube-api-access-p9rn7\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-kube-api-access-p9rn7\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.647961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.647949 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-certs\") pod \"a00341e7-e710-4156-8c80-6947b6d581e7\" (UID: \"a00341e7-e710-4156-8c80-6947b6d581e7\") " Oct 31 15:16:06.649428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.649404 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-certs" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:16:06.649815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.649794 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-credential-socket" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:16:06.652790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.652601 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-envoy" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:16:06.653285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.653014 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00341e7-e710-4156-8c80-6947b6d581e7-istiod-ca-cert" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:16:06.655007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.654978 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-data" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:16:06.655133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.655075 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-socket" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:16:06.655725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.655704 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-istio-token" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:16:06.657094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.657071 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a00341e7-e710-4156-8c80-6947b6d581e7-istio-podinfo" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:16:06.657169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.657152 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-kube-api-access-p9rn7" pod "a00341e7-e710-4156-8c80-6947b6d581e7" (UID: "a00341e7-e710-4156-8c80-6947b6d581e7"). InnerVolumeSpecName "kube-api-access-p9rn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:16:06.750168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750138 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00341e7-e710-4156-8c80-6947b6d581e7-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750167 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750182 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750193 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750203 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00341e7-e710-4156-8c80-6947b6d581e7-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750211 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9rn7\" (UniqueName: \"kubernetes.io/projected/a00341e7-e710-4156-8c80-6947b6d581e7-kube-api-access-p9rn7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750221 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750230 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:06.750370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:06.750238 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00341e7-e710-4156-8c80-6947b6d581e7-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:07.071605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.071572 4865 generic.go:356] "Generic (PLEG): container finished" podID="a00341e7-e710-4156-8c80-6947b6d581e7" containerID="108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c" exitCode=0 Oct 31 15:16:07.072091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.071646 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" Oct 31 15:16:07.072091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.071661 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" event={"ID":"a00341e7-e710-4156-8c80-6947b6d581e7","Type":"ContainerDied","Data":"108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c"} Oct 31 15:16:07.072091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.071744 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl" event={"ID":"a00341e7-e710-4156-8c80-6947b6d581e7","Type":"ContainerDied","Data":"808cfdebf230c23fae3e9af896305a0185872932b6d2e69c86c875c1ac58f191"} Oct 31 15:16:07.072091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.071768 4865 scope.go:122] "RemoveContainer" containerID="108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c" Oct 31 15:16:07.083679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.083656 4865 scope.go:122] "RemoveContainer" containerID="bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a" Oct 31 15:16:07.093018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.093002 4865 scope.go:122] "RemoveContainer" containerID="108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c" Oct 31 15:16:07.093298 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:16:07.093277 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c\": container with ID starting with 108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c not found: ID does not exist" containerID="108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c" Oct 31 15:16:07.093347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.093308 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c"} err="failed to get container status \"108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c\": rpc error: code = NotFound desc = could not find container \"108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c\": container with ID starting with 108fdaaa88883ee7954c7b5191c0b506964e54b7acc0ca437c18298a488b303c not found: ID does not exist" Oct 31 15:16:07.093347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.093327 4865 scope.go:122] "RemoveContainer" containerID="bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a" Oct 31 15:16:07.093756 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:16:07.093737 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a\": container with ID starting with bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a not found: ID does not exist" containerID="bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a" Oct 31 15:16:07.093826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.093764 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a"} err="failed to get container status \"bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a\": rpc error: code = NotFound desc = could not find container \"bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a\": container with ID starting with bafe138a1364440bed32d861b7feef8b585d915a239df1b8d88b0fc71005ca3a not found: ID does not exist" Oct 31 15:16:07.241225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.241196 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl"] Oct 31 15:16:07.309408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:07.309374 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f7251e-openshift-default-864ddcff9f-z5lfl"] Oct 31 15:16:08.021847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:08.021815 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:16:08.498197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:08.498158 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" path="/var/lib/kubelet/pods/a00341e7-e710-4156-8c80-6947b6d581e7/volumes" Oct 31 15:16:08.801600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:08.801528 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:09.051588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:09.051554 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:09.187003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:09.186924 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:11.278656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:11.278625 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:13.029464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:13.029404 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:13.919760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:13.919730 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:14.069123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.069082 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:14.369938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.368422 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:14.485151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.485122 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7"] Oct 31 15:16:14.485933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.485912 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" Oct 31 15:16:14.486013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.485937 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" Oct 31 15:16:14.486122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.486110 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" Oct 31 15:16:14.493309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.493256 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:14.540101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.540063 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt"] Oct 31 15:16:14.541661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.541635 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" Oct 31 15:16:14.541661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.541661 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" Oct 31 15:16:14.541993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.541973 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00341e7-e710-4156-8c80-6947b6d581e7" containerName="istio-proxy" Oct 31 15:16:14.547180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.547154 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.547321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.547178 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7"] Oct 31 15:16:14.597499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.584783 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-9ab5df\"" Oct 31 15:16:14.615342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.614637 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ad5bfe-openshift-default-dockercfg-pg46l\"" Oct 31 15:16:14.634754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.632791 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-9ab5df-dockercfg-l8k2l\"" Oct 31 15:16:14.634975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.618249 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt"] Oct 31 15:16:14.641005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.640859 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:14.650595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.650563 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-certs\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.650890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.650871 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-socket\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.651078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.651065 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-data\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.651202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.651187 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:14.651378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.651346 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7486f\" (UniqueName: \"kubernetes.io/projected/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-kube-api-access-7486f\") pod \"payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:14.651669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.651640 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-podinfo\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.651886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.651868 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-envoy\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.684707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.684672 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istiod-ca-cert\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.685113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.685089 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-credential-socket\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.685268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.685253 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxgz\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-kube-api-access-wzxgz\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.685431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.685407 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-token\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.869307 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-socket\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.869912 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-socket\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870013 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-data\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870062 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870103 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7486f\" (UniqueName: \"kubernetes.io/projected/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-kube-api-access-7486f\") pod \"payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870238 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-podinfo\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870335 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-envoy\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870411 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istiod-ca-cert\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870524 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-credential-socket\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870570 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxgz\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-kube-api-access-wzxgz\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870610 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-token\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.870690 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-certs\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.871659 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-certs\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.872457 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istiod-ca-cert\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.872741 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-credential-socket\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.881815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.878011 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-data\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.903552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.894685 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:14.965311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.965081 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-podinfo\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.972525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.971241 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-token\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.978500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.977452 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-envoy\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:14.978500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.978333 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7486f\" (UniqueName: \"kubernetes.io/projected/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-kube-api-access-7486f\") pod \"payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:14.979008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:14.978982 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxgz\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-kube-api-access-wzxgz\") pod \"e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:15.143499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.143442 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:15.169330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.168002 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:15.182633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.182576 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:15.216937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.216724 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws"] Oct 31 15:16:15.271675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.266132 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc"] Oct 31 15:16:15.512412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.512381 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:15.690540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.685300 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7"] Oct 31 15:16:15.698433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.694814 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt"] Oct 31 15:16:15.698433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.697976 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:15.774577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.774534 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:15.774898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.774871 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:15.774987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:15.774929 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:16.207222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:16.207190 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" event={"ID":"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f","Type":"ContainerStarted","Data":"a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d"} Oct 31 15:16:16.207654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:16.207252 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" event={"ID":"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f","Type":"ContainerStarted","Data":"7e8bd06da791bc287393a33429c09219419fbe04b28e2d17a3e01e4ebde27d7d"} Oct 31 15:16:16.209135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:16.209104 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" event={"ID":"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc","Type":"ContainerStarted","Data":"8712c2a884505f744ea2a3b939754d8ec851491594285b9e081f34e677023890"} Oct 31 15:16:16.209282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:16.209161 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" event={"ID":"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc","Type":"ContainerStarted","Data":"6dd5d98a2a64e7665c27cf11e6dc66bf7b9f6bdafa0a73ac5c12c8bc3597c9ff"} Oct 31 15:16:16.209282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:16.209215 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:16.330771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:16.330722 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podStartSLOduration=2.330707614 podStartE2EDuration="2.330707614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:16.327584715 +0000 UTC m=+1821.947136127" watchObservedRunningTime="2026-10-31 15:16:16.330707614 +0000 UTC m=+1821.950259024" Oct 31 15:16:16.444443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:16.444389 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" podStartSLOduration=2.444375292 podStartE2EDuration="2.444375292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:16.441773081 +0000 UTC m=+1822.061324492" watchObservedRunningTime="2026-10-31 15:16:16.444375292 +0000 UTC m=+1822.063926702" Oct 31 15:16:17.184590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:17.184548 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:17.191274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:17.191248 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:17.217111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:17.216955 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:17.217941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:17.217918 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:16:17.505505 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:16:17.500594 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:16:18.253519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:18.250728 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:20.217757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.217720 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" podUID="6b2ee4de-20dc-4496-98e9-137cf6a2c853" containerName="payload-processing" containerID="cri-o://4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59" gracePeriod=25 Oct 31 15:16:20.242408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.242381 4865 generic.go:356] "Generic (PLEG): container finished" podID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerID="406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d" exitCode=0 Oct 31 15:16:20.242545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.242449 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" event={"ID":"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4","Type":"ContainerDied","Data":"406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d"} Oct 31 15:16:20.242863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.242848 4865 scope.go:122] "RemoveContainer" containerID="406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d" Oct 31 15:16:20.244395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.244361 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:20.244544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.244451 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:20.244617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.244539 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:20.267249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.267221 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" podUID="9843444d-dccd-41e0-b7e4-475e95ea4976" containerName="payload-pre-processing" containerID="cri-o://51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d" gracePeriod=25 Oct 31 15:16:20.487264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.487067 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:20.589524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.589425 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:16:20.658779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.658751 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:16:20.661707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.661682 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:20.693987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.693959 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2ee4de-20dc-4496-98e9-137cf6a2c853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2ee4de-20dc-4496-98e9-137cf6a2c853-plugins-config-volume\") pod \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " Oct 31 15:16:20.694137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.694094 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2ee4de-20dc-4496-98e9-137cf6a2c853-kube-api-access-cfxq6\" (UniqueName: \"kubernetes.io/projected/6b2ee4de-20dc-4496-98e9-137cf6a2c853-kube-api-access-cfxq6\") pod \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\" (UID: \"6b2ee4de-20dc-4496-98e9-137cf6a2c853\") " Oct 31 15:16:20.694377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.694357 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2ee4de-20dc-4496-98e9-137cf6a2c853-plugins-config-volume" pod "6b2ee4de-20dc-4496-98e9-137cf6a2c853" (UID: "6b2ee4de-20dc-4496-98e9-137cf6a2c853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:16:20.694663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.694647 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2ee4de-20dc-4496-98e9-137cf6a2c853-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:20.696543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.696516 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2ee4de-20dc-4496-98e9-137cf6a2c853-kube-api-access-cfxq6" pod "6b2ee4de-20dc-4496-98e9-137cf6a2c853" (UID: "6b2ee4de-20dc-4496-98e9-137cf6a2c853"). InnerVolumeSpecName "kube-api-access-cfxq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:16:20.790933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.790806 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:20.795660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.795639 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9843444d-dccd-41e0-b7e4-475e95ea4976-kube-api-access-m5bqz\" (UniqueName: \"kubernetes.io/projected/9843444d-dccd-41e0-b7e4-475e95ea4976-kube-api-access-m5bqz\") pod \"9843444d-dccd-41e0-b7e4-475e95ea4976\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " Oct 31 15:16:20.795781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.795678 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9843444d-dccd-41e0-b7e4-475e95ea4976-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843444d-dccd-41e0-b7e4-475e95ea4976-plugins-config-volume\") pod \"9843444d-dccd-41e0-b7e4-475e95ea4976\" (UID: \"9843444d-dccd-41e0-b7e4-475e95ea4976\") " Oct 31 15:16:20.796113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.796086 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9843444d-dccd-41e0-b7e4-475e95ea4976-plugins-config-volume" pod "9843444d-dccd-41e0-b7e4-475e95ea4976" (UID: "9843444d-dccd-41e0-b7e4-475e95ea4976"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:16:20.796237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.796119 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfxq6\" (UniqueName: \"kubernetes.io/projected/6b2ee4de-20dc-4496-98e9-137cf6a2c853-kube-api-access-cfxq6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:20.797975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.797951 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9843444d-dccd-41e0-b7e4-475e95ea4976-kube-api-access-m5bqz" pod "9843444d-dccd-41e0-b7e4-475e95ea4976" (UID: "9843444d-dccd-41e0-b7e4-475e95ea4976"). InnerVolumeSpecName "kube-api-access-m5bqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:16:20.921493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.898657 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5bqz\" (UniqueName: \"kubernetes.io/projected/9843444d-dccd-41e0-b7e4-475e95ea4976-kube-api-access-m5bqz\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:20.921689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:20.921598 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9843444d-dccd-41e0-b7e4-475e95ea4976-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:21.250570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.250534 4865 generic.go:356] "Generic (PLEG): container finished" podID="6b2ee4de-20dc-4496-98e9-137cf6a2c853" containerID="4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59" exitCode=0 Oct 31 15:16:21.251061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.250595 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" Oct 31 15:16:21.251061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.250618 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" event={"ID":"6b2ee4de-20dc-4496-98e9-137cf6a2c853","Type":"ContainerDied","Data":"4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59"} Oct 31 15:16:21.251061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.250691 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws" event={"ID":"6b2ee4de-20dc-4496-98e9-137cf6a2c853","Type":"ContainerDied","Data":"f5b48d2534223b19de1fab987bcec6f4403671c62b88c99c83f1dacbad931c78"} Oct 31 15:16:21.251061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.250720 4865 scope.go:122] "RemoveContainer" containerID="4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59" Oct 31 15:16:21.253016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.252990 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" event={"ID":"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4","Type":"ContainerStarted","Data":"435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709"} Oct 31 15:16:21.254559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.254536 4865 generic.go:356] "Generic (PLEG): container finished" podID="9843444d-dccd-41e0-b7e4-475e95ea4976" containerID="51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d" exitCode=0 Oct 31 15:16:21.254715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.254574 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" event={"ID":"9843444d-dccd-41e0-b7e4-475e95ea4976","Type":"ContainerDied","Data":"51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d"} Oct 31 15:16:21.254715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.254604 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" Oct 31 15:16:21.254715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.254612 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc" event={"ID":"9843444d-dccd-41e0-b7e4-475e95ea4976","Type":"ContainerDied","Data":"643eb41131650c6d3d5d2fabe429f73226b2b189b1e7dc1ef33ffdc29647394e"} Oct 31 15:16:21.262624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.262603 4865 scope.go:122] "RemoveContainer" containerID="4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59" Oct 31 15:16:21.262951 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:16:21.262920 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59\": container with ID starting with 4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59 not found: ID does not exist" containerID="4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59" Oct 31 15:16:21.263063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.262964 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59"} err="failed to get container status \"4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59\": rpc error: code = NotFound desc = could not find container \"4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59\": container with ID starting with 4a25cf8a2ec9421e5424e02e2a51722dc5e5ecdacd1d601ecac8ac1dd989ab59 not found: ID does not exist" Oct 31 15:16:21.263063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.262987 4865 scope.go:122] "RemoveContainer" containerID="51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d" Oct 31 15:16:21.273123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.273107 4865 scope.go:122] "RemoveContainer" containerID="51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d" Oct 31 15:16:21.273388 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:16:21.273370 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d\": container with ID starting with 51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d not found: ID does not exist" containerID="51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d" Oct 31 15:16:21.273450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.273395 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d"} err="failed to get container status \"51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d\": rpc error: code = NotFound desc = could not find container \"51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d\": container with ID starting with 51539f694822f4c3c9ba39c725c57ef477e083760c414c8c86dac1546d4a1c1d not found: ID does not exist" Oct 31 15:16:21.290292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.290272 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:16:21.290457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.290438 4865 patch_prober.go:29] interesting pod/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 15:16:21.290554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.290538 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 15:16:21.544052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.544025 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc"] Oct 31 15:16:21.549949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.549926 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-cf65ccf55-44kzc"] Oct 31 15:16:21.663025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.662995 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws"] Oct 31 15:16:21.673869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:21.673838 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-69b955b69c-w6mws"] Oct 31 15:16:22.290149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:22.290115 4865 patch_prober.go:29] interesting pod/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 15:16:22.290595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:22.290188 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 15:16:22.522121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:22.522081 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2ee4de-20dc-4496-98e9-137cf6a2c853" path="/var/lib/kubelet/pods/6b2ee4de-20dc-4496-98e9-137cf6a2c853/volumes" Oct 31 15:16:22.522675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:22.522655 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9843444d-dccd-41e0-b7e4-475e95ea4976" path="/var/lib/kubelet/pods/9843444d-dccd-41e0-b7e4-475e95ea4976/volumes" Oct 31 15:16:23.289927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:23.289894 4865 patch_prober.go:29] interesting pod/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 15:16:23.290081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:23.289968 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 15:16:23.779618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:23.779588 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:23.973488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:23.973432 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:24.293554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.292784 4865 patch_prober.go:29] interesting pod/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 15:16:24.293554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.292875 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 15:16:24.347509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.309826 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl"] Oct 31 15:16:24.347509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.310888 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9843444d-dccd-41e0-b7e4-475e95ea4976" containerName="payload-pre-processing" Oct 31 15:16:24.347509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.310912 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9843444d-dccd-41e0-b7e4-475e95ea4976" containerName="payload-pre-processing" Oct 31 15:16:24.347509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.310946 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6b2ee4de-20dc-4496-98e9-137cf6a2c853" containerName="payload-processing" Oct 31 15:16:24.347509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.310955 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2ee4de-20dc-4496-98e9-137cf6a2c853" containerName="payload-processing" Oct 31 15:16:24.347509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.311132 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9843444d-dccd-41e0-b7e4-475e95ea4976" containerName="payload-pre-processing" Oct 31 15:16:24.347509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.311149 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2ee4de-20dc-4496-98e9-137cf6a2c853" containerName="payload-processing" Oct 31 15:16:24.394755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.392281 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.451145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.447692 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad5bfe-dockercfg-8h6hf\"" Oct 31 15:16:24.451145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.447801 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad5bfe\"" Oct 31 15:16:24.451145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.449330 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdr6j\" (UniqueName: \"kubernetes.io/projected/ac41e17e-b7f0-42c7-8900-08150a0365f5-kube-api-access-pdr6j\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.451145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.449432 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac41e17e-b7f0-42c7-8900-08150a0365f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.451705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.451628 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl"] Oct 31 15:16:24.479455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.479421 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw"] Oct 31 15:16:24.577999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.577901 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac41e17e-b7f0-42c7-8900-08150a0365f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.578199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.578157 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdr6j\" (UniqueName: \"kubernetes.io/projected/ac41e17e-b7f0-42c7-8900-08150a0365f5-kube-api-access-pdr6j\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.585454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.581147 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac41e17e-b7f0-42c7-8900-08150a0365f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.624494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.620791 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw"] Oct 31 15:16:24.624494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.621585 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:24.636044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.636022 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdr6j\" (UniqueName: \"kubernetes.io/projected/ac41e17e-b7f0-42c7-8900-08150a0365f5-kube-api-access-pdr6j\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.762194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.762156 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:24.780216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.780179 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:24.780645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.780444 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2gc\" (UniqueName: \"kubernetes.io/projected/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-kube-api-access-rl2gc\") pod \"payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:24.882183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.882136 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2gc\" (UniqueName: \"kubernetes.io/projected/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-kube-api-access-rl2gc\") pod \"payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:24.882436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.882412 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:24.883399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.883373 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:24.941917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:24.941885 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2gc\" (UniqueName: \"kubernetes.io/projected/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-kube-api-access-rl2gc\") pod \"payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:25.138364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.138262 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:25.181976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.181768 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl"] Oct 31 15:16:25.250792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.250766 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:25.290187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.289851 4865 patch_prober.go:29] interesting pod/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 15:16:25.290187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.289893 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 15:16:25.434707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.433643 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" event={"ID":"ac41e17e-b7f0-42c7-8900-08150a0365f5","Type":"ContainerStarted","Data":"0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76"} Oct 31 15:16:25.434707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.433695 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" event={"ID":"ac41e17e-b7f0-42c7-8900-08150a0365f5","Type":"ContainerStarted","Data":"49e227634f97bc6e594f1b45f3ec8b50c5928927d36214cd117d273f3ce30223"} Oct 31 15:16:25.434707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.434568 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:25.553737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.553680 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" podStartSLOduration=1.553661483 podStartE2EDuration="1.553661483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:25.550655335 +0000 UTC m=+1831.170206746" watchObservedRunningTime="2026-10-31 15:16:25.553661483 +0000 UTC m=+1831.173212895" Oct 31 15:16:25.653232 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:16:25.653203 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33665aab_7efa_45e3_9e3c_f6dd9f4ef091.slice/crio-aa63787fa7cfcb7dbd019269996c9677fac51cd83ade4ec62cebe4910c767ae1 WatchSource:0}: Error finding container aa63787fa7cfcb7dbd019269996c9677fac51cd83ade4ec62cebe4910c767ae1: Status 404 returned error can't find the container with id aa63787fa7cfcb7dbd019269996c9677fac51cd83ade4ec62cebe4910c767ae1 Oct 31 15:16:25.653625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:25.653605 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw"] Oct 31 15:16:26.289918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:26.289883 4865 patch_prober.go:29] interesting pod/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 15:16:26.290370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:26.289940 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 15:16:26.441741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:26.441709 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" event={"ID":"33665aab-7efa-45e3-9e3c-f6dd9f4ef091","Type":"ContainerStarted","Data":"9b2dfb98fb2675ea006893192c395b0931171ba08c54cd61ec6d49760d8ba8ea"} Oct 31 15:16:26.441940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:26.441770 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" event={"ID":"33665aab-7efa-45e3-9e3c-f6dd9f4ef091","Type":"ContainerStarted","Data":"aa63787fa7cfcb7dbd019269996c9677fac51cd83ade4ec62cebe4910c767ae1"} Oct 31 15:16:26.441940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:26.441802 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:26.563880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:26.563757 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" podStartSLOduration=2.563741989 podStartE2EDuration="2.563741989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:26.561233468 +0000 UTC m=+1832.180784904" watchObservedRunningTime="2026-10-31 15:16:26.563741989 +0000 UTC m=+1832.183293452" Oct 31 15:16:27.219583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:27.219556 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:16:27.294519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:27.294486 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:16:27.294928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:27.294740 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:16:27.295410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:27.295393 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:16:29.793188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:29.793149 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:34.691179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:34.691143 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:36.501401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:36.501324 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:37.452017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:37.451990 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:16:37.452218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:37.452065 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:16:38.143066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:38.143032 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:38.404926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:38.404847 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh"] Oct 31 15:16:38.520234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:38.520202 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8"] Oct 31 15:16:38.765539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:38.765508 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:38.911532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:38.911496 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:41.517225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:41.517191 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:43.405418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.405377 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" podUID="e617b98e-2f12-4415-99fd-64f97d7f0ccc" containerName="payload-processing" containerID="cri-o://f81f154243e3f9318d6b014e6976351488d00a40f854dd60f2409aad0d96913c" gracePeriod=25 Oct 31 15:16:43.521482 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.521427 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" podUID="c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" containerName="payload-pre-processing" containerID="cri-o://1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df" gracePeriod=25 Oct 31 15:16:43.561681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.561645 4865 generic.go:356] "Generic (PLEG): container finished" podID="e617b98e-2f12-4415-99fd-64f97d7f0ccc" containerID="f81f154243e3f9318d6b014e6976351488d00a40f854dd60f2409aad0d96913c" exitCode=0 Oct 31 15:16:43.561828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.561720 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" event={"ID":"e617b98e-2f12-4415-99fd-64f97d7f0ccc","Type":"ContainerDied","Data":"f81f154243e3f9318d6b014e6976351488d00a40f854dd60f2409aad0d96913c"} Oct 31 15:16:43.688285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.688006 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:16:43.811756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.811728 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e617b98e-2f12-4415-99fd-64f97d7f0ccc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617b98e-2f12-4415-99fd-64f97d7f0ccc-plugins-config-volume\") pod \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " Oct 31 15:16:43.811947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.811815 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e617b98e-2f12-4415-99fd-64f97d7f0ccc-kube-api-access-q6c8b\" (UniqueName: \"kubernetes.io/projected/e617b98e-2f12-4415-99fd-64f97d7f0ccc-kube-api-access-q6c8b\") pod \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\" (UID: \"e617b98e-2f12-4415-99fd-64f97d7f0ccc\") " Oct 31 15:16:43.812071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.812048 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e617b98e-2f12-4415-99fd-64f97d7f0ccc-plugins-config-volume" pod "e617b98e-2f12-4415-99fd-64f97d7f0ccc" (UID: "e617b98e-2f12-4415-99fd-64f97d7f0ccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:16:43.812330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.812314 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e617b98e-2f12-4415-99fd-64f97d7f0ccc-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:43.814523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.814498 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e617b98e-2f12-4415-99fd-64f97d7f0ccc-kube-api-access-q6c8b" pod "e617b98e-2f12-4415-99fd-64f97d7f0ccc" (UID: "e617b98e-2f12-4415-99fd-64f97d7f0ccc"). InnerVolumeSpecName "kube-api-access-q6c8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:16:43.828640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.828612 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:16:43.913196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:43.913121 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6c8b\" (UniqueName: \"kubernetes.io/projected/e617b98e-2f12-4415-99fd-64f97d7f0ccc-kube-api-access-q6c8b\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:44.014123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.014088 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-kube-api-access-rg4jv\" (UniqueName: \"kubernetes.io/projected/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-kube-api-access-rg4jv\") pod \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " Oct 31 15:16:44.014329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.014240 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-plugins-config-volume\") pod \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\" (UID: \"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9\") " Oct 31 15:16:44.014691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.014665 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-plugins-config-volume" pod "c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" (UID: "c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:16:44.014902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.014886 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:44.016304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.016282 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-kube-api-access-rg4jv" pod "c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" (UID: "c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9"). InnerVolumeSpecName "kube-api-access-rg4jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:16:44.116411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.116369 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4jv\" (UniqueName: \"kubernetes.io/projected/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9-kube-api-access-rg4jv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:16:44.270836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.270797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:44.570081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.570012 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" Oct 31 15:16:44.570081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.570024 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh" event={"ID":"e617b98e-2f12-4415-99fd-64f97d7f0ccc","Type":"ContainerDied","Data":"78eb741992ab1235c05f972834623bfbfed4a0ce10bc77eed5bc5cf1f760339f"} Oct 31 15:16:44.570596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.570091 4865 scope.go:122] "RemoveContainer" containerID="f81f154243e3f9318d6b014e6976351488d00a40f854dd60f2409aad0d96913c" Oct 31 15:16:44.571513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.571436 4865 generic.go:356] "Generic (PLEG): container finished" podID="c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" containerID="1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df" exitCode=0 Oct 31 15:16:44.571630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.571516 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" event={"ID":"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9","Type":"ContainerDied","Data":"1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df"} Oct 31 15:16:44.571630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.571533 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" Oct 31 15:16:44.571630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.571610 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8" event={"ID":"c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9","Type":"ContainerDied","Data":"ea4786e161c5f39ea31744025db110af8c1fa1b22bc565dd33b2fcb2f3cda225"} Oct 31 15:16:44.580386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.580369 4865 scope.go:122] "RemoveContainer" containerID="1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df" Oct 31 15:16:44.589442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.589424 4865 scope.go:122] "RemoveContainer" containerID="1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df" Oct 31 15:16:44.589763 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:16:44.589743 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df\": container with ID starting with 1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df not found: ID does not exist" containerID="1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df" Oct 31 15:16:44.589814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.589771 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df"} err="failed to get container status \"1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df\": rpc error: code = NotFound desc = could not find container \"1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df\": container with ID starting with 1c49a65852a69db84602be5cb1a11aaa9691c4f187718f5fbb4fdd73399930df not found: ID does not exist" Oct 31 15:16:44.738370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.738331 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh"] Oct 31 15:16:44.789054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.789023 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a538f1-5b4456bf4c-gsnrh"] Oct 31 15:16:44.900893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.900806 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8"] Oct 31 15:16:44.903884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:44.903863 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a538f1-57c5565687-9l5z8"] Oct 31 15:16:46.487109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:46.487063 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:46.497337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:46.497310 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" path="/var/lib/kubelet/pods/c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9/volumes" Oct 31 15:16:46.497824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:46.497809 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e617b98e-2f12-4415-99fd-64f97d7f0ccc" path="/var/lib/kubelet/pods/e617b98e-2f12-4415-99fd-64f97d7f0ccc/volumes" Oct 31 15:16:47.491076 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:16:47.491038 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:16:48.579795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:48.579759 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:48.800314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:48.800280 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:48.932880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:48.932799 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:49.357398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.357359 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:49.854824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.854796 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm"] Oct 31 15:16:49.855452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.855437 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" containerName="payload-pre-processing" Oct 31 15:16:49.855549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.855454 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" containerName="payload-pre-processing" Oct 31 15:16:49.855549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.855520 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e617b98e-2f12-4415-99fd-64f97d7f0ccc" containerName="payload-processing" Oct 31 15:16:49.855549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.855527 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e617b98e-2f12-4415-99fd-64f97d7f0ccc" containerName="payload-processing" Oct 31 15:16:49.855656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.855648 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e617b98e-2f12-4415-99fd-64f97d7f0ccc" containerName="payload-processing" Oct 31 15:16:49.855693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.855664 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a4bdc2-b7bd-4a14-937c-96f5cfca8cc9" containerName="payload-pre-processing" Oct 31 15:16:49.860205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.860187 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908411 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-podinfo\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908513 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-data\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908553 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-certs\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908641 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-token\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908684 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d091acc-c288-43e4-99dd-a505a4f62fb0-istiod-ca-cert\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908808 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-envoy\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908845 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpglh\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-kube-api-access-xpglh\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908894 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-credential-socket\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.912495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.908960 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-socket\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:49.936497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.925161 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9cc557-openshift-default-dockercfg-b5wcm\"" Oct 31 15:16:49.997498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:49.987003 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043491 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-token\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043567 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d091acc-c288-43e4-99dd-a505a4f62fb0-istiod-ca-cert\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043683 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-envoy\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043718 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpglh\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-kube-api-access-xpglh\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043763 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-credential-socket\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043841 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-socket\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043944 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-podinfo\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.043998 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-data\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.044032 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-certs\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.044612 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-certs\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.045053 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-credential-socket\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.045138 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-socket\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.045707 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d091acc-c288-43e4-99dd-a505a4f62fb0-istiod-ca-cert\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.075214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.046756 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-data\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.093583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.093550 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm"] Oct 31 15:16:50.157594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.126446 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-podinfo\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.157594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.138461 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-envoy\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.171038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.171003 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-token\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.175012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.174978 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpglh\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-kube-api-access-xpglh\") pod \"e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.197519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.197430 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:50.198521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.198441 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:50.693317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.690361 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm"] Oct 31 15:16:50.703009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.702952 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:50.703281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.703251 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:50.703594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:50.703567 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:16:51.642827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:51.642795 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" event={"ID":"5d091acc-c288-43e4-99dd-a505a4f62fb0","Type":"ContainerStarted","Data":"54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f"} Oct 31 15:16:51.643263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:51.642855 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" event={"ID":"5d091acc-c288-43e4-99dd-a505a4f62fb0","Type":"ContainerStarted","Data":"02c1b4d76db8b9f6c76a981973458f0f4a483dc79db8b30fe3aea95df495c71a"} Oct 31 15:16:51.747138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:51.747103 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:52.199337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:52.199312 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:52.204701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:52.204681 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:52.331604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:52.331552 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podStartSLOduration=3.331536334 podStartE2EDuration="3.331536334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:51.829508167 +0000 UTC m=+1857.449059591" watchObservedRunningTime="2026-10-31 15:16:52.331536334 +0000 UTC m=+1857.951087746" Oct 31 15:16:52.650226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:52.650195 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:52.651145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:52.651121 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:16:54.710668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:54.710636 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:54.942825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:54.942797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:55.076609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:55.076539 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:55.229607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:55.229578 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:55.371803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:55.371720 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:56.611358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:56.611323 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:56.651585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:56.651551 4865 scope.go:122] "RemoveContainer" containerID="05d9b8738b2a52ae9ae963aea7d58a2976db6711d8ddbbf08604bcefd91f2d5b" Oct 31 15:16:56.739515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:56.739456 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:57.915638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:57.915604 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:58.202499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.202406 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:16:58.635429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.635395 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh"] Oct 31 15:16:58.649698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.649670 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh"] Oct 31 15:16:58.649902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.649793 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:58.703068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.703034 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9cc557\"" Oct 31 15:16:58.754430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.754395 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9cc557-dockercfg-sqqsk\"" Oct 31 15:16:58.787961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.787933 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47j4\" (UniqueName: \"kubernetes.io/projected/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-kube-api-access-z47j4\") pod \"payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:58.788130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.787979 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:58.889894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.889803 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47j4\" (UniqueName: \"kubernetes.io/projected/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-kube-api-access-z47j4\") pod \"payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:58.889894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.889857 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:58.890637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.890617 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:58.947997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.947964 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47j4\" (UniqueName: \"kubernetes.io/projected/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-kube-api-access-z47j4\") pod \"payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:58.967868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:58.967836 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:59.350918 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:16:59.350884 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920b0860_9a3f_4fa0_a26d_5aca8ef4bd15.slice/crio-f2d33f5eff650f1b470a56daf259563c203f5a76ecfdc8a2774a00db78373706 WatchSource:0}: Error finding container f2d33f5eff650f1b470a56daf259563c203f5a76ecfdc8a2774a00db78373706: Status 404 returned error can't find the container with id f2d33f5eff650f1b470a56daf259563c203f5a76ecfdc8a2774a00db78373706 Oct 31 15:16:59.351922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:59.351899 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh"] Oct 31 15:16:59.705324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:59.705222 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" event={"ID":"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15","Type":"ContainerStarted","Data":"3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4"} Oct 31 15:16:59.705509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:59.705412 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:16:59.705509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:59.705440 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" event={"ID":"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15","Type":"ContainerStarted","Data":"f2d33f5eff650f1b470a56daf259563c203f5a76ecfdc8a2774a00db78373706"} Oct 31 15:16:59.831154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:16:59.831081 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" podStartSLOduration=1.8310606759999999 podStartE2EDuration="1.831060676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:59.830391533 +0000 UTC m=+1865.449942946" watchObservedRunningTime="2026-10-31 15:16:59.831060676 +0000 UTC m=+1865.450612087" Oct 31 15:17:01.499868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:01.499832 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:01.865851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:01.865757 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:17:06.216765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:06.216730 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:07.985616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:07.985581 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:08.123949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:08.123916 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:09.545821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:09.545790 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w"] Oct 31 15:17:09.546295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:09.545974 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" containerID="cri-o://c3e7e14e871833775063950fc0fec8eb178c676d72b08e96cbdbf11e8fb3d14b" gracePeriod=30 Oct 31 15:17:10.713975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:10.713946 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:17:10.967199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:10.967118 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6"] Oct 31 15:17:10.967387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:10.967361 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" containerID="cri-o://0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa" gracePeriod=30 Oct 31 15:17:11.097435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:11.097401 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:11.115213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:11.115167 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5_9474b299-b315-4569-b444-801ff9376570/payload-processing/0.log" Oct 31 15:17:11.723603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:11.723570 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:17:11.723995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:11.723635 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:17:11.993223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:11.993149 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq_6230a2fd-0f3d-486c-b501-7228db16112e/payload-processing/0.log" Oct 31 15:17:12.192972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:12.192945 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq_6230a2fd-0f3d-486c-b501-7228db16112e/payload-processing/0.log" Oct 31 15:17:13.099544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:13.099506 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:14.599786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:14.599758 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:14.842589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:14.842245 4865 generic.go:356] "Generic (PLEG): container finished" podID="24fae054-db19-4628-9430-283e5164ebdb" containerID="c3e7e14e871833775063950fc0fec8eb178c676d72b08e96cbdbf11e8fb3d14b" exitCode=0 Oct 31 15:17:14.842589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:14.842357 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" event={"ID":"24fae054-db19-4628-9430-283e5164ebdb","Type":"ContainerDied","Data":"c3e7e14e871833775063950fc0fec8eb178c676d72b08e96cbdbf11e8fb3d14b"} Oct 31 15:17:14.973020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:14.972998 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:17:15.070691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.070660 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24fae054-db19-4628-9430-283e5164ebdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fae054-db19-4628-9430-283e5164ebdb-istiod-ca-cert\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.070859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.070730 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-credential-socket\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.070859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.070797 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-kube-api-access-vfwnx\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-kube-api-access-vfwnx\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.070989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.070860 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24fae054-db19-4628-9430-283e5164ebdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fae054-db19-4628-9430-283e5164ebdb-istio-podinfo\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.070989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.070908 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-data\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.070989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.070936 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-istio-token\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-istio-token\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.071169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.071002 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-envoy\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.071169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.070997 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fae054-db19-4628-9430-283e5164ebdb-istiod-ca-cert" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:15.071169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.071054 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-credential-socket" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:15.071355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.071176 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-certs\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.071355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.071207 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-data" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:15.071355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.071213 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-socket\") pod \"24fae054-db19-4628-9430-283e5164ebdb\" (UID: \"24fae054-db19-4628-9430-283e5164ebdb\") " Oct 31 15:17:15.071610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.071403 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-certs" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:15.071610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.071420 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-socket" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:15.072078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.072053 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.072238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.072221 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.072360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.072337 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fae054-db19-4628-9430-283e5164ebdb-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.072519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.072363 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.072519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.072379 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.074009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.073985 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-envoy" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:15.074113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.074037 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-kube-api-access-vfwnx" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "kube-api-access-vfwnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:15.074240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.074220 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24fae054-db19-4628-9430-283e5164ebdb-istio-podinfo" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:17:15.074385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.074357 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-istio-token" pod "24fae054-db19-4628-9430-283e5164ebdb" (UID: "24fae054-db19-4628-9430-283e5164ebdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:15.089092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.089062 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:15.200620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.174456 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwnx\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-kube-api-access-vfwnx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.200620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.195167 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fae054-db19-4628-9430-283e5164ebdb-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.200620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.195188 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fae054-db19-4628-9430-283e5164ebdb-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.200620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.195202 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fae054-db19-4628-9430-283e5164ebdb-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:15.850409 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.850382 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" Oct 31 15:17:15.850409 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.850392 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w" event={"ID":"24fae054-db19-4628-9430-283e5164ebdb","Type":"ContainerDied","Data":"3f68c1e051267a3a1ad8ef9890b397304a03f316d5b7ae5fbbdb8edf4b923231"} Oct 31 15:17:15.850935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:15.850446 4865 scope.go:122] "RemoveContainer" containerID="c3e7e14e871833775063950fc0fec8eb178c676d72b08e96cbdbf11e8fb3d14b" Oct 31 15:17:16.071650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.071618 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w"] Oct 31 15:17:16.125230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.125149 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sgf9w"] Oct 31 15:17:16.194053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.194026 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv"] Oct 31 15:17:16.194900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.194878 4865 cpu_manager.go:414] "containerMap: removing container" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" Oct 31 15:17:16.195019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.194901 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" Oct 31 15:17:16.195124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.195110 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fae054-db19-4628-9430-283e5164ebdb" containerName="istio-proxy" Oct 31 15:17:16.198661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.198643 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.248483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.248452 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:17:16.252456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.252435 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-032b05-dockercfg-mwj9s\"" Oct 31 15:17:16.252964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.252941 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-032b05\"" Oct 31 15:17:16.256169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.256150 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv"] Oct 31 15:17:16.327150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327116 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-socket\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327160 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-credential-socket\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327256 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-data\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327284 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-certs\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327324 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88a516a3-4e43-461a-adef-41d03726eec5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a516a3-4e43-461a-adef-41d03726eec5-istio-podinfo\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327377 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-socket" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:16.327507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327415 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a516a3-4e43-461a-adef-41d03726eec5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a516a3-4e43-461a-adef-41d03726eec5-istiod-ca-cert\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327504 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-istio-token\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-istio-token\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327486 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-credential-socket" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:16.327668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327627 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-kube-api-access-bvwqw\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-kube-api-access-bvwqw\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327547 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-certs" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:16.327887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327568 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-data" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:16.327887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327699 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-envoy\") pod \"88a516a3-4e43-461a-adef-41d03726eec5\" (UID: \"88a516a3-4e43-461a-adef-41d03726eec5\") " Oct 31 15:17:16.327887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.327808 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a516a3-4e43-461a-adef-41d03726eec5-istiod-ca-cert" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:16.328420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.328324 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.328516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.328431 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mln\" (UniqueName: \"kubernetes.io/projected/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-kube-api-access-65mln\") pod \"payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.328688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.328669 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.328762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.328694 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.328762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.328708 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.328762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.328722 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.328762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.328736 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a516a3-4e43-461a-adef-41d03726eec5-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.329748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.329730 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-istio-token" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:16.329882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.329867 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-envoy" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:16.330295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.330273 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88a516a3-4e43-461a-adef-41d03726eec5-istio-podinfo" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:17:16.330295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.330291 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-kube-api-access-bvwqw" pod "88a516a3-4e43-461a-adef-41d03726eec5" (UID: "88a516a3-4e43-461a-adef-41d03726eec5"). InnerVolumeSpecName "kube-api-access-bvwqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:16.430503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.430408 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.430742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.430721 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mln\" (UniqueName: \"kubernetes.io/projected/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-kube-api-access-65mln\") pod \"payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.431209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.431187 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.431361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.431347 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvwqw\" (UniqueName: \"kubernetes.io/projected/88a516a3-4e43-461a-adef-41d03726eec5-kube-api-access-bvwqw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.431508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.431495 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a516a3-4e43-461a-adef-41d03726eec5-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.431607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.431596 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a516a3-4e43-461a-adef-41d03726eec5-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:16.431675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.431228 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.488896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.488866 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mln\" (UniqueName: \"kubernetes.io/projected/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-kube-api-access-65mln\") pod \"payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.499153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.499122 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24fae054-db19-4628-9430-283e5164ebdb" path="/var/lib/kubelet/pods/24fae054-db19-4628-9430-283e5164ebdb/volumes" Oct 31 15:17:16.515735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.515717 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:16.859078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.859046 4865 generic.go:356] "Generic (PLEG): container finished" podID="88a516a3-4e43-461a-adef-41d03726eec5" containerID="0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa" exitCode=0 Oct 31 15:17:16.859571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.859098 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" Oct 31 15:17:16.859571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.859138 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" event={"ID":"88a516a3-4e43-461a-adef-41d03726eec5","Type":"ContainerDied","Data":"0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa"} Oct 31 15:17:16.859571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.859190 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6" event={"ID":"88a516a3-4e43-461a-adef-41d03726eec5","Type":"ContainerDied","Data":"8a48b77036a1607f46d738b64a20586fbc00cc48a02c5835eae8bbd8497c75e0"} Oct 31 15:17:16.859571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.859214 4865 scope.go:122] "RemoveContainer" containerID="0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa" Oct 31 15:17:16.869511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.869487 4865 scope.go:122] "RemoveContainer" containerID="0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa" Oct 31 15:17:16.869797 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:16.869779 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa\": container with ID starting with 0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa not found: ID does not exist" containerID="0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa" Oct 31 15:17:16.869849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.869805 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa"} err="failed to get container status \"0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa\": rpc error: code = NotFound desc = could not find container \"0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa\": container with ID starting with 0e5d235c4f67ab6450f1898babc4aa68a33ef5103cc45a91a316397edb6e4ffa not found: ID does not exist" Oct 31 15:17:16.902689 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:17:16.902669 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171ca979_9d95_4ead_b4a6_f6a5ff8e14b9.slice/crio-17abfdcc7322909d4349c7363403809dba4ecadff135d6c61c4f7224fe9e1296 WatchSource:0}: Error finding container 17abfdcc7322909d4349c7363403809dba4ecadff135d6c61c4f7224fe9e1296: Status 404 returned error can't find the container with id 17abfdcc7322909d4349c7363403809dba4ecadff135d6c61c4f7224fe9e1296 Oct 31 15:17:16.902846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.902828 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv"] Oct 31 15:17:16.980014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.979991 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh"] Oct 31 15:17:16.980921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.980904 4865 cpu_manager.go:414] "containerMap: removing container" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" Oct 31 15:17:16.980984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.980924 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" Oct 31 15:17:16.981065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.981053 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a516a3-4e43-461a-adef-41d03726eec5" containerName="istio-proxy" Oct 31 15:17:16.985747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:16.985730 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.029151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.029125 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6"] Oct 31 15:17:17.032142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.032121 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-qgkd6"] Oct 31 15:17:17.041320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.041287 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a20fe9f-e754-480f-b136-70ce4beee1c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.041530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.041511 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pllz7\" (UniqueName: \"kubernetes.io/projected/7a20fe9f-e754-480f-b136-70ce4beee1c3-kube-api-access-pllz7\") pod \"payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.080342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.080313 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh"] Oct 31 15:17:17.143421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.143341 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pllz7\" (UniqueName: \"kubernetes.io/projected/7a20fe9f-e754-480f-b136-70ce4beee1c3-kube-api-access-pllz7\") pod \"payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.143636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.143535 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a20fe9f-e754-480f-b136-70ce4beee1c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.144216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.144193 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a20fe9f-e754-480f-b136-70ce4beee1c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.206204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.206170 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pllz7\" (UniqueName: \"kubernetes.io/projected/7a20fe9f-e754-480f-b136-70ce4beee1c3-kube-api-access-pllz7\") pod \"payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.300451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.300419 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.490708 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:17.490672 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:17:17.559048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.559002 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:17.695882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.695836 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh"] Oct 31 15:17:17.722287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.722257 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:17.870898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.870864 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" event={"ID":"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9","Type":"ContainerStarted","Data":"94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742"} Oct 31 15:17:17.871304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.870930 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" event={"ID":"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9","Type":"ContainerStarted","Data":"17abfdcc7322909d4349c7363403809dba4ecadff135d6c61c4f7224fe9e1296"} Oct 31 15:17:17.871304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.870964 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:17.872712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.872680 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" event={"ID":"7a20fe9f-e754-480f-b136-70ce4beee1c3","Type":"ContainerStarted","Data":"34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131"} Oct 31 15:17:17.872810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.872735 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" event={"ID":"7a20fe9f-e754-480f-b136-70ce4beee1c3","Type":"ContainerStarted","Data":"6a2ceecb68a6bb0b78f54e957ff283630542aab7f600256f31acfbb10416493d"} Oct 31 15:17:17.872810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.872776 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:17.883953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.883927 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:17.989721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:17.989676 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" podStartSLOduration=1.989662999 podStartE2EDuration="1.989662999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:17:17.987555994 +0000 UTC m=+1883.607107406" watchObservedRunningTime="2026-10-31 15:17:17.989662999 +0000 UTC m=+1883.609214442" Oct 31 15:17:18.069566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:18.069495 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv"] Oct 31 15:17:18.105255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:18.105184 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" podStartSLOduration=2.105163782 podStartE2EDuration="2.105163782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:17:18.102626202 +0000 UTC m=+1883.722177614" watchObservedRunningTime="2026-10-31 15:17:18.105163782 +0000 UTC m=+1883.724715195" Oct 31 15:17:18.163109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:18.163081 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:18.501340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:18.501304 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a516a3-4e43-461a-adef-41d03726eec5" path="/var/lib/kubelet/pods/88a516a3-4e43-461a-adef-41d03726eec5/volumes" Oct 31 15:17:21.826655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:21.826566 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:22.251505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.251427 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl"] Oct 31 15:17:22.251752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.251725 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" podUID="16b7708f-f497-4077-8a99-460a5ac8e5da" containerName="payload-processing" containerID="cri-o://ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4" gracePeriod=30 Oct 31 15:17:22.333483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.333448 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:22.461867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.461060 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:22.540049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.540023 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:17:22.659768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.659738 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b7708f-f497-4077-8a99-460a5ac8e5da-kube-api-access-sjvsg\" (UniqueName: \"kubernetes.io/projected/16b7708f-f497-4077-8a99-460a5ac8e5da-kube-api-access-sjvsg\") pod \"16b7708f-f497-4077-8a99-460a5ac8e5da\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " Oct 31 15:17:22.659970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.659951 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b7708f-f497-4077-8a99-460a5ac8e5da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b7708f-f497-4077-8a99-460a5ac8e5da-plugins-config-volume\") pod \"16b7708f-f497-4077-8a99-460a5ac8e5da\" (UID: \"16b7708f-f497-4077-8a99-460a5ac8e5da\") " Oct 31 15:17:22.660319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.660292 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b7708f-f497-4077-8a99-460a5ac8e5da-plugins-config-volume" pod "16b7708f-f497-4077-8a99-460a5ac8e5da" (UID: "16b7708f-f497-4077-8a99-460a5ac8e5da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:22.660561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.660536 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b7708f-f497-4077-8a99-460a5ac8e5da-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.662692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.662667 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b7708f-f497-4077-8a99-460a5ac8e5da-kube-api-access-sjvsg" pod "16b7708f-f497-4077-8a99-460a5ac8e5da" (UID: "16b7708f-f497-4077-8a99-460a5ac8e5da"). InnerVolumeSpecName "kube-api-access-sjvsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:22.800631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.769125 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvsg\" (UniqueName: \"kubernetes.io/projected/16b7708f-f497-4077-8a99-460a5ac8e5da-kube-api-access-sjvsg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.909401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.909369 4865 generic.go:356] "Generic (PLEG): container finished" podID="16b7708f-f497-4077-8a99-460a5ac8e5da" containerID="ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4" exitCode=0 Oct 31 15:17:22.909845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.909420 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" Oct 31 15:17:22.909845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.909464 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" event={"ID":"16b7708f-f497-4077-8a99-460a5ac8e5da","Type":"ContainerDied","Data":"ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4"} Oct 31 15:17:22.909845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.909554 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl" event={"ID":"16b7708f-f497-4077-8a99-460a5ac8e5da","Type":"ContainerDied","Data":"c19d5b3daa5d295090672e517d071c6865164e4c20b64c2697cad18a63e95bb8"} Oct 31 15:17:22.909845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.909576 4865 scope.go:122] "RemoveContainer" containerID="ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4" Oct 31 15:17:22.920254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.920238 4865 scope.go:122] "RemoveContainer" containerID="ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4" Oct 31 15:17:22.920557 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:22.920535 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4\": container with ID starting with ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4 not found: ID does not exist" containerID="ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4" Oct 31 15:17:22.920652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:22.920563 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4"} err="failed to get container status \"ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4\": rpc error: code = NotFound desc = could not find container \"ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4\": container with ID starting with ecf08c99a836a6915a0f43bdb2e3fc1c8c4884e18d4af02fa2f8f0c9115847e4 not found: ID does not exist" Oct 31 15:17:23.077030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:23.076930 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl"] Oct 31 15:17:23.080032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:23.080010 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2196e4-5986d87d76-5qdfl"] Oct 31 15:17:23.489019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:23.488987 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:23.650969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:23.650939 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:23.798574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:23.794018 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:24.498555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:24.498523 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b7708f-f497-4077-8a99-460a5ac8e5da" path="/var/lib/kubelet/pods/16b7708f-f497-4077-8a99-460a5ac8e5da/volumes" Oct 31 15:17:25.941264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:25.941229 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:26.072243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:26.072206 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:27.588218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:27.588181 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:28.086079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:28.086047 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:28.100289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:28.100261 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5"] Oct 31 15:17:28.100509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:28.100484 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" containerID="cri-o://4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca" gracePeriod=30 Oct 31 15:17:28.584652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:28.584623 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:28.864282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:28.864198 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:28.881651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:28.881622 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:17:28.882505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:28.882487 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:17:32.374627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:32.374586 4865 patch_prober.go:29] interesting pod/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:17:32.375025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:32.374650 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:17:33.091710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.091669 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" podUID="0620fcf9-acc1-44b0-9329-963bb2b8f55d" containerName="main" containerID="cri-o://a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe" gracePeriod=45 Oct 31 15:17:33.391166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.391144 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:17:33.408363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.408342 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:17:33.451493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451447 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-home\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-home\") pod \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " Oct 31 15:17:33.451592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451573 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0620fcf9-acc1-44b0-9329-963bb2b8f55d-kube-api-access-9b2gd\" (UniqueName: \"kubernetes.io/projected/0620fcf9-acc1-44b0-9329-963bb2b8f55d-kube-api-access-9b2gd\") pod \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " Oct 31 15:17:33.451679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451663 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-model-cache\") pod \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " Oct 31 15:17:33.451715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451675 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-home" pod "0620fcf9-acc1-44b0-9329-963bb2b8f55d" (UID: "0620fcf9-acc1-44b0-9329-963bb2b8f55d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.451764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451753 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-dshm\") pod \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " Oct 31 15:17:33.451816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451804 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tmp-dir\") pod \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " Oct 31 15:17:33.451866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451856 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tls-certs\") pod \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\" (UID: \"0620fcf9-acc1-44b0-9329-963bb2b8f55d\") " Oct 31 15:17:33.451927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.451905 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-model-cache" pod "0620fcf9-acc1-44b0-9329-963bb2b8f55d" (UID: "0620fcf9-acc1-44b0-9329-963bb2b8f55d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.452120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.452099 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tmp-dir" pod "0620fcf9-acc1-44b0-9329-963bb2b8f55d" (UID: "0620fcf9-acc1-44b0-9329-963bb2b8f55d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.452364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.452344 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.452364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.452365 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.452735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.452377 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.453974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.453951 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-dshm" pod "0620fcf9-acc1-44b0-9329-963bb2b8f55d" (UID: "0620fcf9-acc1-44b0-9329-963bb2b8f55d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.453974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.453962 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tls-certs" pod "0620fcf9-acc1-44b0-9329-963bb2b8f55d" (UID: "0620fcf9-acc1-44b0-9329-963bb2b8f55d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:17:33.454097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.453972 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0620fcf9-acc1-44b0-9329-963bb2b8f55d-kube-api-access-9b2gd" pod "0620fcf9-acc1-44b0-9329-963bb2b8f55d" (UID: "0620fcf9-acc1-44b0-9329-963bb2b8f55d"). InnerVolumeSpecName "kube-api-access-9b2gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:33.553701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.553665 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-socket\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.553701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.553702 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-kube-api-access-pzw7x\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-kube-api-access-pzw7x\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.553962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.553789 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-credential-socket\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.553962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.553816 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-certs\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.554068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.553983 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-socket" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.554131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554103 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-credential-socket" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.554198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554168 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-certs" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.554277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554249 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e480e53c-7fff-4aad-92f7-c52755b17294-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e480e53c-7fff-4aad-92f7-c52755b17294-istiod-ca-cert\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.554341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554280 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e480e53c-7fff-4aad-92f7-c52755b17294-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e480e53c-7fff-4aad-92f7-c52755b17294-istio-podinfo\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.554422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554399 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-data\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.554499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554443 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-envoy\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.554573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554540 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-istio-token\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-istio-token\") pod \"e480e53c-7fff-4aad-92f7-c52755b17294\" (UID: \"e480e53c-7fff-4aad-92f7-c52755b17294\") " Oct 31 15:17:33.554666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554643 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-data" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.554666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.554651 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e480e53c-7fff-4aad-92f7-c52755b17294-istiod-ca-cert" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:33.555664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.555644 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.555873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.555855 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b2gd\" (UniqueName: \"kubernetes.io/projected/0620fcf9-acc1-44b0-9329-963bb2b8f55d-kube-api-access-9b2gd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.555995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.555970 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.555995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.555996 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0620fcf9-acc1-44b0-9329-963bb2b8f55d-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.556189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.556011 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.556189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.556025 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.556189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.556040 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0620fcf9-acc1-44b0-9329-963bb2b8f55d-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.556189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.556054 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e480e53c-7fff-4aad-92f7-c52755b17294-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.556948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.556914 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-kube-api-access-pzw7x" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "kube-api-access-pzw7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:33.557181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.557158 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-envoy" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:33.557386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.557369 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-istio-token" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:33.557856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.557836 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e480e53c-7fff-4aad-92f7-c52755b17294-istio-podinfo" pod "e480e53c-7fff-4aad-92f7-c52755b17294" (UID: "e480e53c-7fff-4aad-92f7-c52755b17294"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:17:33.657091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.657016 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e480e53c-7fff-4aad-92f7-c52755b17294-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.657091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.657043 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e480e53c-7fff-4aad-92f7-c52755b17294-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.657091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.657053 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.657091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.657062 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzw7x\" (UniqueName: \"kubernetes.io/projected/e480e53c-7fff-4aad-92f7-c52755b17294-kube-api-access-pzw7x\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:33.985544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.985509 4865 generic.go:356] "Generic (PLEG): container finished" podID="0620fcf9-acc1-44b0-9329-963bb2b8f55d" containerID="a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe" exitCode=0 Oct 31 15:17:33.985544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.985541 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" Oct 31 15:17:33.985806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.985560 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" event={"ID":"0620fcf9-acc1-44b0-9329-963bb2b8f55d","Type":"ContainerDied","Data":"a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe"} Oct 31 15:17:33.985806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.985606 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv" event={"ID":"0620fcf9-acc1-44b0-9329-963bb2b8f55d","Type":"ContainerDied","Data":"8f0e680c22e229ff299a5c75335120e317f6744589a4593caa57f2bd5ba729da"} Oct 31 15:17:33.985806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.985625 4865 scope.go:122] "RemoveContainer" containerID="a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe" Oct 31 15:17:33.988203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.988180 4865 generic.go:356] "Generic (PLEG): container finished" podID="e480e53c-7fff-4aad-92f7-c52755b17294" containerID="4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca" exitCode=0 Oct 31 15:17:33.988322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.988285 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" Oct 31 15:17:33.988322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.988300 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" event={"ID":"e480e53c-7fff-4aad-92f7-c52755b17294","Type":"ContainerDied","Data":"4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca"} Oct 31 15:17:33.988442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.988348 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5" event={"ID":"e480e53c-7fff-4aad-92f7-c52755b17294","Type":"ContainerDied","Data":"2e9e452031b165d54c28efc5a93ec1db43dbce089187f9337267e680391e46da"} Oct 31 15:17:33.996646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.996598 4865 scope.go:122] "RemoveContainer" containerID="a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe" Oct 31 15:17:33.996921 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:33.996897 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe\": container with ID starting with a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe not found: ID does not exist" containerID="a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe" Oct 31 15:17:33.997001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.996929 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe"} err="failed to get container status \"a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe\": rpc error: code = NotFound desc = could not find container \"a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe\": container with ID starting with a5f3af74a47602434bfedcb7608286a7c2fd6de88a04dcf737cf243c1ab8a1fe not found: ID does not exist" Oct 31 15:17:33.997001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:33.996951 4865 scope.go:122] "RemoveContainer" containerID="4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca" Oct 31 15:17:34.006299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.006283 4865 scope.go:122] "RemoveContainer" containerID="1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec" Oct 31 15:17:34.015039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.015023 4865 scope.go:122] "RemoveContainer" containerID="4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca" Oct 31 15:17:34.015315 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:34.015296 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca\": container with ID starting with 4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca not found: ID does not exist" containerID="4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca" Oct 31 15:17:34.015401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.015324 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca"} err="failed to get container status \"4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca\": rpc error: code = NotFound desc = could not find container \"4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca\": container with ID starting with 4bd1ce566cf3fcbe08009bf830db1c574c73183ff1e8a6d22058d9276881d3ca not found: ID does not exist" Oct 31 15:17:34.015401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.015339 4865 scope.go:122] "RemoveContainer" containerID="1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec" Oct 31 15:17:34.015629 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:34.015607 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec\": container with ID starting with 1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec not found: ID does not exist" containerID="1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec" Oct 31 15:17:34.015676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.015637 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec"} err="failed to get container status \"1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec\": rpc error: code = NotFound desc = could not find container \"1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec\": container with ID starting with 1a4c4c00bb936269156edd09739e5341165e08fba6f7e824479713197a59e4ec not found: ID does not exist" Oct 31 15:17:34.153170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.153138 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv"] Oct 31 15:17:34.156494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.156460 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-644a8a/ipp-route-644a8a-kserve-585f96bb65-jkxlv"] Oct 31 15:17:34.267451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.267289 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5"] Oct 31 15:17:34.270744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.270722 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2196e4-openshift-default-645b649877-vzvt5"] Oct 31 15:17:34.499884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.499845 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0620fcf9-acc1-44b0-9329-963bb2b8f55d" path="/var/lib/kubelet/pods/0620fcf9-acc1-44b0-9329-963bb2b8f55d/volumes" Oct 31 15:17:34.500546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.500525 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" path="/var/lib/kubelet/pods/e480e53c-7fff-4aad-92f7-c52755b17294/volumes" Oct 31 15:17:34.579284 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:34.579201 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:35.136493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:35.134381 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:38.153052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:38.153018 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:17:39.421495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:39.421108 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:41.318496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:41.317646 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:47.490837 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:47.490795 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:17:47.653515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:47.653401 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:48.258621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:48.258584 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:17:48.258792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:48.258653 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:17:49.084696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.084656 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:49.132217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.132187 4865 generic.go:356] "Generic (PLEG): container finished" podID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerID="a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d" exitCode=0 Oct 31 15:17:49.132404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.132237 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" event={"ID":"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f","Type":"ContainerDied","Data":"a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d"} Oct 31 15:17:49.143269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.132780 4865 scope.go:122] "RemoveContainer" containerID="a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d" Oct 31 15:17:49.143269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.139381 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:17:49.143269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.139510 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:17:49.143269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.139561 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:17:49.217720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:49.217684 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:50.141248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:50.141216 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" event={"ID":"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f","Type":"ContainerStarted","Data":"f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151"} Oct 31 15:17:50.183956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:50.183925 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:17:50.184140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:50.184106 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 15:17:50.184184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:50.184149 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 15:17:50.652701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:50.652647 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:51.140111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:51.137245 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:51.143784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:51.143361 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr"] Oct 31 15:17:51.143784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:51.143648 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" containerID="cri-o://9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da" gracePeriod=30 Oct 31 15:17:51.183783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:51.183752 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 15:17:51.183952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:51.183806 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 15:17:52.184731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:52.184698 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 15:17:52.185162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:52.184757 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 15:17:52.654975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:52.654940 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:52.781228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:52.781195 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:53.184728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.184694 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 15:17:53.184922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.184748 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.263495 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z"] Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264439 4865 cpu_manager.go:414] "containerMap: removing container" podUID="16b7708f-f497-4077-8a99-460a5ac8e5da" containerName="payload-processing" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264459 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b7708f-f497-4077-8a99-460a5ac8e5da" containerName="payload-processing" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264583 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264595 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264631 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264640 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264670 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0620fcf9-acc1-44b0-9329-963bb2b8f55d" containerName="main" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264678 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0620fcf9-acc1-44b0-9329-963bb2b8f55d" containerName="main" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264893 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b7708f-f497-4077-8a99-460a5ac8e5da" containerName="payload-processing" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264919 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264938 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e480e53c-7fff-4aad-92f7-c52755b17294" containerName="istio-proxy" Oct 31 15:17:53.266510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.264952 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0620fcf9-acc1-44b0-9329-963bb2b8f55d" containerName="main" Oct 31 15:17:53.329528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.314801 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:53.329528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.318979 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.488498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.488439 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-certs\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.488766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.488739 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-data\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.488933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.488915 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15e1799-085c-409b-8365-6f8ea4cdeda1-istiod-ca-cert\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.489059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.489044 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-socket\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.489203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.489187 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-credential-socket\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.489344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.489329 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pb2s\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-kube-api-access-9pb2s\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.489450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.489437 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-token\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.489737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.489718 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-podinfo\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.489874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.489858 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-envoy\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.491194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.491170 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-379d72-openshift-default-dockercfg-rknh6\"" Oct 31 15:17:53.536495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.525622 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z"] Oct 31 15:17:53.590916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.590887 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-certs\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.590916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.590918 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-data\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.590938 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15e1799-085c-409b-8365-6f8ea4cdeda1-istiod-ca-cert\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.590960 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-socket\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591233 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-credential-socket\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591260 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-certs\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591292 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pb2s\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-kube-api-access-9pb2s\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591317 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-token\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591419 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-data\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591450 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-socket\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591494 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-podinfo\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591539 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-envoy\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.591704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.591660 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-credential-socket\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.592215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.592197 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15e1799-085c-409b-8365-6f8ea4cdeda1-istiod-ca-cert\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.625613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.625579 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-podinfo\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.625968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.625940 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-envoy\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.665037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.665001 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:53.674793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.671106 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pb2s\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-kube-api-access-9pb2s\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.679247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.679213 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-token\") pod \"e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:53.866357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:53.866269 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:54.184399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:54.184325 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 15:17:54.184399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:54.184376 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 15:17:54.323778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:54.323747 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z"] Oct 31 15:17:54.325893 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:17:54.325863 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15e1799_085c_409b_8365_6f8ea4cdeda1.slice/crio-e593fee532bcb32915d8a47940aa43fe7fd6fd155c9d2b6c6f91f7e7d6ddad7c WatchSource:0}: Error finding container e593fee532bcb32915d8a47940aa43fe7fd6fd155c9d2b6c6f91f7e7d6ddad7c: Status 404 returned error can't find the container with id e593fee532bcb32915d8a47940aa43fe7fd6fd155c9d2b6c6f91f7e7d6ddad7c Oct 31 15:17:54.328986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:54.328951 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:17:54.329125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:54.329020 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:17:54.329125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:54.329050 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:17:55.184027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.183994 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 15:17:55.184224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.184070 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 15:17:55.187238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.187199 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" event={"ID":"e15e1799-085c-409b-8365-6f8ea4cdeda1","Type":"ContainerStarted","Data":"da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524"} Oct 31 15:17:55.187364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.187271 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" event={"ID":"e15e1799-085c-409b-8365-6f8ea4cdeda1","Type":"ContainerStarted","Data":"e593fee532bcb32915d8a47940aa43fe7fd6fd155c9d2b6c6f91f7e7d6ddad7c"} Oct 31 15:17:55.306796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.306766 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:55.416493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.415011 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podStartSLOduration=2.414987467 podStartE2EDuration="2.414987467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:17:55.361994172 +0000 UTC m=+1920.981545584" watchObservedRunningTime="2026-10-31 15:17:55.414987467 +0000 UTC m=+1921.034538880" Oct 31 15:17:55.505103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.505041 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7"] Oct 31 15:17:55.556031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.556003 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.576598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.576546 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:55.599177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.599136 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7"] Oct 31 15:17:55.615792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.615600 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:17:55.615792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.615606 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:17:55.615792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.615719 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:17:55.615792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.615741 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:17:55.661782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.661741 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09517863-9826-4062-93a6-742cce83ab09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.661976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.661919 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09517863-9826-4062-93a6-742cce83ab09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.662039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.661984 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km8xw\" (UniqueName: \"kubernetes.io/projected/09517863-9826-4062-93a6-742cce83ab09-kube-api-access-km8xw\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.763243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.763152 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09517863-9826-4062-93a6-742cce83ab09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.763427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.763282 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09517863-9826-4062-93a6-742cce83ab09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.763427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.763364 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km8xw\" (UniqueName: \"kubernetes.io/projected/09517863-9826-4062-93a6-742cce83ab09-kube-api-access-km8xw\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.764134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.764102 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09517863-9826-4062-93a6-742cce83ab09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.766553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.766523 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09517863-9826-4062-93a6-742cce83ab09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.822195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.822153 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km8xw\" (UniqueName: \"kubernetes.io/projected/09517863-9826-4062-93a6-742cce83ab09-kube-api-access-km8xw\") pod \"maas-api-revoke-sub-609721ee-89lr7\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:55.857957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.857926 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:55.867161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.867139 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:55.873054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.873034 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:55.905671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:55.905649 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:56.188616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.188593 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:17:56.188959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.188938 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:17:56.189894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.189878 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:17:56.194909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.194889 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:56.195595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.195578 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:17:56.294768 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:17:56.294736 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09517863_9826_4062_93a6_742cce83ab09.slice/crio-25335b9c83b824ff22cf2e94ea140fbd60544ab46663df73c9caa6b01383c4d0 WatchSource:0}: Error finding container 25335b9c83b824ff22cf2e94ea140fbd60544ab46663df73c9caa6b01383c4d0: Status 404 returned error can't find the container with id 25335b9c83b824ff22cf2e94ea140fbd60544ab46663df73c9caa6b01383c4d0 Oct 31 15:17:56.295956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.295927 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7"] Oct 31 15:17:56.412765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.412711 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574532 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-socket\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574586 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-podinfo\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574630 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-data\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574663 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-token\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-token\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574724 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-kube-api-access-s88xv\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-kube-api-access-s88xv\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574801 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-certs\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574862 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-credential-socket\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574903 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a588319-ac1c-4ff5-af1a-e7902e68b263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a588319-ac1c-4ff5-af1a-e7902e68b263-istiod-ca-cert\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.576491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.574959 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-envoy\") pod \"9a588319-ac1c-4ff5-af1a-e7902e68b263\" (UID: \"9a588319-ac1c-4ff5-af1a-e7902e68b263\") " Oct 31 15:17:56.578430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.578400 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-data" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:56.578651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.578619 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-credential-socket" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:56.578786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.578758 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-certs" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:56.579326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.579181 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a588319-ac1c-4ff5-af1a-e7902e68b263-istiod-ca-cert" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:56.579547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.579525 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-socket" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:56.580041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.580016 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-envoy" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:17:56.582278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.582246 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-kube-api-access-s88xv" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "kube-api-access-s88xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:56.582448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.582424 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-podinfo" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:17:56.587843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.587822 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-token" pod "9a588319-ac1c-4ff5-af1a-e7902e68b263" (UID: "9a588319-ac1c-4ff5-af1a-e7902e68b263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:56.677133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677056 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677087 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a588319-ac1c-4ff5-af1a-e7902e68b263-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677100 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677114 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677126 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677139 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677151 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677162 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88xv\" (UniqueName: \"kubernetes.io/projected/9a588319-ac1c-4ff5-af1a-e7902e68b263-kube-api-access-s88xv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.677428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.677174 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a588319-ac1c-4ff5-af1a-e7902e68b263-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:56.964156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:56.964121 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:57.204140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.204108 4865 generic.go:356] "Generic (PLEG): container finished" podID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerID="9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da" exitCode=0 Oct 31 15:17:57.204327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.204161 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" Oct 31 15:17:57.204327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.204187 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" event={"ID":"9a588319-ac1c-4ff5-af1a-e7902e68b263","Type":"ContainerDied","Data":"9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da"} Oct 31 15:17:57.204327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.204248 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr" event={"ID":"9a588319-ac1c-4ff5-af1a-e7902e68b263","Type":"ContainerDied","Data":"71fb2c457f88d6ff8d10e969a17f9246e6421c08d19147c50e0b081b682b2962"} Oct 31 15:17:57.204327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.204284 4865 scope.go:122] "RemoveContainer" containerID="9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da" Oct 31 15:17:57.206607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.206242 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" event={"ID":"09517863-9826-4062-93a6-742cce83ab09","Type":"ContainerStarted","Data":"e0d9f28010a7ba5982f4efd481dc2d12dbade904b4e83cce59c64626bdc29d90"} Oct 31 15:17:57.206607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.206293 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" event={"ID":"09517863-9826-4062-93a6-742cce83ab09","Type":"ContainerStarted","Data":"25335b9c83b824ff22cf2e94ea140fbd60544ab46663df73c9caa6b01383c4d0"} Oct 31 15:17:57.215404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.215384 4865 scope.go:122] "RemoveContainer" containerID="0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681" Oct 31 15:17:57.224865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.224850 4865 scope.go:122] "RemoveContainer" containerID="9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da" Oct 31 15:17:57.225154 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:57.225136 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da\": container with ID starting with 9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da not found: ID does not exist" containerID="9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da" Oct 31 15:17:57.225207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.225163 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da"} err="failed to get container status \"9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da\": rpc error: code = NotFound desc = could not find container \"9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da\": container with ID starting with 9d23ef52cd992bf431948b3fbbc37eb5893f8ff71376a79d1735834ef42261da not found: ID does not exist" Oct 31 15:17:57.225207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.225179 4865 scope.go:122] "RemoveContainer" containerID="0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681" Oct 31 15:17:57.225403 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:17:57.225384 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681\": container with ID starting with 0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681 not found: ID does not exist" containerID="0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681" Oct 31 15:17:57.225449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.225409 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681"} err="failed to get container status \"0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681\": rpc error: code = NotFound desc = could not find container \"0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681\": container with ID starting with 0a11605b694fff65dd00ba0430dba3b096cdcfe6fa5652760af3803491224681 not found: ID does not exist" Oct 31 15:17:57.372972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.372945 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr"] Oct 31 15:17:57.423598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.423564 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a538f1-openshift-default-64cf866ddf-zc7vr"] Oct 31 15:17:57.490492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.490380 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" podStartSLOduration=2.490366616 podStartE2EDuration="2.490366616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:17:57.487912587 +0000 UTC m=+1923.107464021" watchObservedRunningTime="2026-10-31 15:17:57.490366616 +0000 UTC m=+1923.109918026" Oct 31 15:17:57.917628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:57.917550 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:58.154414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:58.154380 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:58.215060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:58.215031 4865 generic.go:356] "Generic (PLEG): container finished" podID="09517863-9826-4062-93a6-742cce83ab09" containerID="e0d9f28010a7ba5982f4efd481dc2d12dbade904b4e83cce59c64626bdc29d90" exitCode=0 Oct 31 15:17:58.215259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:58.215108 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" event={"ID":"09517863-9826-4062-93a6-742cce83ab09","Type":"ContainerDied","Data":"e0d9f28010a7ba5982f4efd481dc2d12dbade904b4e83cce59c64626bdc29d90"} Oct 31 15:17:58.430861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:58.430829 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:58.504557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:58.504463 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" path="/var/lib/kubelet/pods/9a588319-ac1c-4ff5-af1a-e7902e68b263/volumes" Oct 31 15:17:58.588024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:58.587997 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:17:59.381418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.381391 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:17:59.516754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.516668 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09517863-9826-4062-93a6-742cce83ab09-kube-api-access-km8xw\" (UniqueName: \"kubernetes.io/projected/09517863-9826-4062-93a6-742cce83ab09-kube-api-access-km8xw\") pod \"09517863-9826-4062-93a6-742cce83ab09\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " Oct 31 15:17:59.516929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.516818 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09517863-9826-4062-93a6-742cce83ab09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09517863-9826-4062-93a6-742cce83ab09-maas-api-openshift-ca\") pod \"09517863-9826-4062-93a6-742cce83ab09\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " Oct 31 15:17:59.517015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.516994 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09517863-9826-4062-93a6-742cce83ab09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09517863-9826-4062-93a6-742cce83ab09-maas-api-xks-ca\") pod \"09517863-9826-4062-93a6-742cce83ab09\" (UID: \"09517863-9826-4062-93a6-742cce83ab09\") " Oct 31 15:17:59.517137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.517112 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09517863-9826-4062-93a6-742cce83ab09-maas-api-openshift-ca" pod "09517863-9826-4062-93a6-742cce83ab09" (UID: "09517863-9826-4062-93a6-742cce83ab09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:59.517525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.517493 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09517863-9826-4062-93a6-742cce83ab09-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:59.522533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.519670 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09517863-9826-4062-93a6-742cce83ab09-maas-api-xks-ca" pod "09517863-9826-4062-93a6-742cce83ab09" (UID: "09517863-9826-4062-93a6-742cce83ab09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:17:59.522533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.519883 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09517863-9826-4062-93a6-742cce83ab09-kube-api-access-km8xw" pod "09517863-9826-4062-93a6-742cce83ab09" (UID: "09517863-9826-4062-93a6-742cce83ab09"). InnerVolumeSpecName "kube-api-access-km8xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:59.619367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.619339 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09517863-9826-4062-93a6-742cce83ab09-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:59.619367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:17:59.619365 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km8xw\" (UniqueName: \"kubernetes.io/projected/09517863-9826-4062-93a6-742cce83ab09-kube-api-access-km8xw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:00.229991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:00.229964 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" Oct 31 15:18:00.230166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:00.229962 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7" event={"ID":"09517863-9826-4062-93a6-742cce83ab09","Type":"ContainerDied","Data":"25335b9c83b824ff22cf2e94ea140fbd60544ab46663df73c9caa6b01383c4d0"} Oct 31 15:18:00.230166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:00.230085 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25335b9c83b824ff22cf2e94ea140fbd60544ab46663df73c9caa6b01383c4d0" Oct 31 15:18:01.581512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:01.581453 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:01.976822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:01.976788 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:02.149331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.149296 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:02.389026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.388931 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7"] Oct 31 15:18:02.390221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.390195 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" Oct 31 15:18:02.390370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.390360 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" Oct 31 15:18:02.390434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.390427 4865 cpu_manager.go:414] "containerMap: removing container" podUID="09517863-9826-4062-93a6-742cce83ab09" containerName="revoke-keys" Oct 31 15:18:02.390513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.390503 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09517863-9826-4062-93a6-742cce83ab09" containerName="revoke-keys" Oct 31 15:18:02.390722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.390712 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" Oct 31 15:18:02.390790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.390783 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="09517863-9826-4062-93a6-742cce83ab09" containerName="revoke-keys" Oct 31 15:18:02.390841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.390835 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" Oct 31 15:18:02.394738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.394712 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.454393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.454363 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7"] Oct 31 15:18:02.460910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.460867 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189fd5aa-4825-426d-bfa3-f601a4aaa71b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.461391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.461354 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlwm\" (UniqueName: \"kubernetes.io/projected/189fd5aa-4825-426d-bfa3-f601a4aaa71b-kube-api-access-7nlwm\") pod \"payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.508797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.508765 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-379d72-dockercfg-frc5c\"" Oct 31 15:18:02.530493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.517396 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-379d72\"" Oct 31 15:18:02.601527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.586638 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189fd5aa-4825-426d-bfa3-f601a4aaa71b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.601527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.586868 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlwm\" (UniqueName: \"kubernetes.io/projected/189fd5aa-4825-426d-bfa3-f601a4aaa71b-kube-api-access-7nlwm\") pod \"payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.627457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.623702 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189fd5aa-4825-426d-bfa3-f601a4aaa71b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.670188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.670086 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm"] Oct 31 15:18:02.671044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.670976 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" Oct 31 15:18:02.671044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.671001 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a588319-ac1c-4ff5-af1a-e7902e68b263" containerName="istio-proxy" Oct 31 15:18:02.683225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.683198 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlwm\" (UniqueName: \"kubernetes.io/projected/189fd5aa-4825-426d-bfa3-f601a4aaa71b-kube-api-access-7nlwm\") pod \"payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.688690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.688658 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm"] Oct 31 15:18:02.688803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.688791 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:02.710711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.710631 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:02.792637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.792599 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdmg\" (UniqueName: \"kubernetes.io/projected/c8505626-7c66-4416-90c5-177d9ca407e5-kube-api-access-fjdmg\") pod \"payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:02.792944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.792924 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8505626-7c66-4416-90c5-177d9ca407e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:02.894566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.894535 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdmg\" (UniqueName: \"kubernetes.io/projected/c8505626-7c66-4416-90c5-177d9ca407e5-kube-api-access-fjdmg\") pod \"payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:02.894771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.894720 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8505626-7c66-4416-90c5-177d9ca407e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:02.895401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.895379 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8505626-7c66-4416-90c5-177d9ca407e5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:02.952421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:02.952354 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdmg\" (UniqueName: \"kubernetes.io/projected/c8505626-7c66-4416-90c5-177d9ca407e5-kube-api-access-fjdmg\") pod \"payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:03.019531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.019500 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:03.090967 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:18:03.090922 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189fd5aa_4825_426d_bfa3_f601a4aaa71b.slice/crio-920b59612b8ee5d40c64fd3d12bd1f49c876fea6b70669b247f0c3ee65ea8e7a WatchSource:0}: Error finding container 920b59612b8ee5d40c64fd3d12bd1f49c876fea6b70669b247f0c3ee65ea8e7a: Status 404 returned error can't find the container with id 920b59612b8ee5d40c64fd3d12bd1f49c876fea6b70669b247f0c3ee65ea8e7a Oct 31 15:18:03.091560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.091282 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7"] Oct 31 15:18:03.239579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.239542 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:03.254274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.254247 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" event={"ID":"189fd5aa-4825-426d-bfa3-f601a4aaa71b","Type":"ContainerStarted","Data":"a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc"} Oct 31 15:18:03.254418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.254299 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" event={"ID":"189fd5aa-4825-426d-bfa3-f601a4aaa71b","Type":"ContainerStarted","Data":"920b59612b8ee5d40c64fd3d12bd1f49c876fea6b70669b247f0c3ee65ea8e7a"} Oct 31 15:18:03.254418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.254392 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:03.385662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.385609 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" podStartSLOduration=1.385595004 podStartE2EDuration="1.385595004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:18:03.383416463 +0000 UTC m=+1929.002967873" watchObservedRunningTime="2026-10-31 15:18:03.385595004 +0000 UTC m=+1929.005146414" Oct 31 15:18:03.404498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.404443 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:03.405724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.405695 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm"] Oct 31 15:18:03.406410 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:18:03.406387 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8505626_7c66_4416_90c5_177d9ca407e5.slice/crio-a0422b2c95f9cc5ae0ff80ffb3f5604c4cdf2365168692868a42f309e293f4e6 WatchSource:0}: Error finding container a0422b2c95f9cc5ae0ff80ffb3f5604c4cdf2365168692868a42f309e293f4e6: Status 404 returned error can't find the container with id a0422b2c95f9cc5ae0ff80ffb3f5604c4cdf2365168692868a42f309e293f4e6 Oct 31 15:18:03.545552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:03.536929 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:04.262447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:04.262411 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" event={"ID":"c8505626-7c66-4416-90c5-177d9ca407e5","Type":"ContainerStarted","Data":"632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937"} Oct 31 15:18:04.262942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:04.262495 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" event={"ID":"c8505626-7c66-4416-90c5-177d9ca407e5","Type":"ContainerStarted","Data":"a0422b2c95f9cc5ae0ff80ffb3f5604c4cdf2365168692868a42f309e293f4e6"} Oct 31 15:18:04.262942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:04.262694 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:04.380330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:04.380268 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" podStartSLOduration=2.380254308 podStartE2EDuration="2.380254308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:18:04.377575418 +0000 UTC m=+1929.997126829" watchObservedRunningTime="2026-10-31 15:18:04.380254308 +0000 UTC m=+1929.999805719" Oct 31 15:18:07.134917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:07.134843 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:07.606106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:07.606073 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:18:08.217115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:08.217084 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:13.064660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:13.064590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:14.265861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:14.265830 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:18:15.282889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:15.282861 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:18:17.491409 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:18:17.491364 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:18:17.910116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:17.909991 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:22.630790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:22.630739 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:28.935103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:28.935065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:33.619294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:33.619261 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:36.085668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:36.085624 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:36.237791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:36.237759 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:36.373418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:36.373340 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:37.515742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:37.515676 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:38.029394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.029039 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:38.186700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.186652 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:38.462552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.456749 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5"] Oct 31 15:18:38.462552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.457071 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" podUID="9474b299-b315-4569-b444-801ff9376570" containerName="payload-processing" containerID="cri-o://0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4" gracePeriod=30 Oct 31 15:18:38.600540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.555202 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:38.763717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.763676 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:38.841625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.841601 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:18:38.924280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.924251 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9474b299-b315-4569-b444-801ff9376570-kube-api-access-z2x4m\" (UniqueName: \"kubernetes.io/projected/9474b299-b315-4569-b444-801ff9376570-kube-api-access-z2x4m\") pod \"9474b299-b315-4569-b444-801ff9376570\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " Oct 31 15:18:38.924458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.924341 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9474b299-b315-4569-b444-801ff9376570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9474b299-b315-4569-b444-801ff9376570-plugins-config-volume\") pod \"9474b299-b315-4569-b444-801ff9376570\" (UID: \"9474b299-b315-4569-b444-801ff9376570\") " Oct 31 15:18:38.924742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.924706 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9474b299-b315-4569-b444-801ff9376570-plugins-config-volume" pod "9474b299-b315-4569-b444-801ff9376570" (UID: "9474b299-b315-4569-b444-801ff9376570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:38.925264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.925240 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9474b299-b315-4569-b444-801ff9376570-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:38.926821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:38.926793 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9474b299-b315-4569-b444-801ff9376570-kube-api-access-z2x4m" pod "9474b299-b315-4569-b444-801ff9376570" (UID: "9474b299-b315-4569-b444-801ff9376570"). InnerVolumeSpecName "kube-api-access-z2x4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:39.040574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.034773 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2x4m\" (UniqueName: \"kubernetes.io/projected/9474b299-b315-4569-b444-801ff9376570-kube-api-access-z2x4m\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:39.529224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.529178 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:39.634150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.634105 4865 generic.go:356] "Generic (PLEG): container finished" podID="9474b299-b315-4569-b444-801ff9376570" containerID="0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4" exitCode=0 Oct 31 15:18:39.634693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.634166 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" Oct 31 15:18:39.634693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.634229 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" event={"ID":"9474b299-b315-4569-b444-801ff9376570","Type":"ContainerDied","Data":"0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4"} Oct 31 15:18:39.634693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.634290 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5" event={"ID":"9474b299-b315-4569-b444-801ff9376570","Type":"ContainerDied","Data":"40fd19a9914530e05b185170f9c041f351e7d850d44cc83cfe8c26fb927443c3"} Oct 31 15:18:39.634693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.634314 4865 scope.go:122] "RemoveContainer" containerID="0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4" Oct 31 15:18:39.647645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.647622 4865 scope.go:122] "RemoveContainer" containerID="0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4" Oct 31 15:18:39.647938 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:18:39.647905 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4\": container with ID starting with 0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4 not found: ID does not exist" containerID="0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4" Oct 31 15:18:39.648015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.647945 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4"} err="failed to get container status \"0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4\": rpc error: code = NotFound desc = could not find container \"0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4\": container with ID starting with 0801c5eab6a6c3d647ce1004035e6249ac5e32db816aab9ab69b47e665b56fd4 not found: ID does not exist" Oct 31 15:18:39.669603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.669577 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:39.810644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.810555 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5"] Oct 31 15:18:39.853726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.853693 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-644a8a-75b966b87d-m2lg5"] Oct 31 15:18:39.977426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:39.977392 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:40.125549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:40.124598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:40.497699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:40.497673 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9474b299-b315-4569-b444-801ff9376570" path="/var/lib/kubelet/pods/9474b299-b315-4569-b444-801ff9376570/volumes" Oct 31 15:18:40.817241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:40.817157 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:18:41.055941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:41.055911 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:41.331509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:41.331461 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:41.797636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:41.797601 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:41.817072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:41.817047 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh"] Oct 31 15:18:41.817276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:41.817253 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" containerID="cri-o://628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3" gracePeriod=30 Oct 31 15:18:42.565500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:42.565436 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:42.603709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:42.603682 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw"] Oct 31 15:18:42.712462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:42.712429 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl"] Oct 31 15:18:42.951585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:42.951515 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:43.216500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:43.214833 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:43.391007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:43.390978 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:43.565143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:43.565066 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:43.694400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:43.694370 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:45.354808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:45.354777 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:45.708631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:45.708599 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:46.565061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:46.565030 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:46.783831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:46.783798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:46.987080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:46.987053 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl"] Oct 31 15:18:46.987958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:46.987937 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9474b299-b315-4569-b444-801ff9376570" containerName="payload-processing" Oct 31 15:18:46.988011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:46.987962 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9474b299-b315-4569-b444-801ff9376570" containerName="payload-processing" Oct 31 15:18:46.988190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:46.988166 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9474b299-b315-4569-b444-801ff9376570" containerName="payload-processing" Oct 31 15:18:46.991900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:46.991880 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.116552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.099791 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b62069-dockercfg-qlc4n\"" Oct 31 15:18:47.116552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.099891 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b62069\"" Oct 31 15:18:47.178845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.170542 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb9xj\" (UniqueName: \"kubernetes.io/projected/d6e233da-d5cd-41d0-8b04-af6edc905ca8-kube-api-access-lb9xj\") pod \"payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.179158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.179128 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e233da-d5cd-41d0-8b04-af6edc905ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.207565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.207529 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl"] Oct 31 15:18:47.242672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.242453 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq"] Oct 31 15:18:47.246813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.246790 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq"] Oct 31 15:18:47.246979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.246911 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.280960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.280934 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb9xj\" (UniqueName: \"kubernetes.io/projected/d6e233da-d5cd-41d0-8b04-af6edc905ca8-kube-api-access-lb9xj\") pod \"payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.281112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.280980 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e233da-d5cd-41d0-8b04-af6edc905ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.282100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.282074 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e233da-d5cd-41d0-8b04-af6edc905ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.295099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.295078 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:18:47.339940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.339905 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb9xj\" (UniqueName: \"kubernetes.io/projected/d6e233da-d5cd-41d0-8b04-af6edc905ca8-kube-api-access-lb9xj\") pod \"payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.382406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382315 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-data\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382447 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-token\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-token\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382546 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-kube-api-access-8mfvn\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-kube-api-access-8mfvn\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382577 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-certs\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382620 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-podinfo\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382682 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-credential-socket\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382727 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-envoy\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382762 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2adb8cf5-05ef-49af-ac31-714008b9f57c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adb8cf5-05ef-49af-ac31-714008b9f57c-istiod-ca-cert\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.382818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382815 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-socket\") pod \"2adb8cf5-05ef-49af-ac31-714008b9f57c\" (UID: \"2adb8cf5-05ef-49af-ac31-714008b9f57c\") " Oct 31 15:18:47.383096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.382954 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-certs" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.383265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383239 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df4b69-5198-4a06-8df2-8ea612beca27-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.383403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383207 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-data" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.383504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383445 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9f7\" (UniqueName: \"kubernetes.io/projected/33df4b69-5198-4a06-8df2-8ea612beca27-kube-api-access-sh9f7\") pod \"payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.383579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383512 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-credential-socket" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.383742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383720 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.383742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383725 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-socket" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.383869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383748 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adb8cf5-05ef-49af-ac31-714008b9f57c-istiod-ca-cert" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:47.383869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383782 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.383869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.383801 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.385992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.385968 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-token" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:47.386237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.386212 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-podinfo" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:18:47.386766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.386738 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-kube-api-access-8mfvn" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "kube-api-access-8mfvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:47.387256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.387231 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-envoy" pod "2adb8cf5-05ef-49af-ac31-714008b9f57c" (UID: "2adb8cf5-05ef-49af-ac31-714008b9f57c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.484713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.484677 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df4b69-5198-4a06-8df2-8ea612beca27-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.484903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.484841 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9f7\" (UniqueName: \"kubernetes.io/projected/33df4b69-5198-4a06-8df2-8ea612beca27-kube-api-access-sh9f7\") pod \"payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.485071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.485057 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.485147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.485077 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfvn\" (UniqueName: \"kubernetes.io/projected/2adb8cf5-05ef-49af-ac31-714008b9f57c-kube-api-access-8mfvn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.485147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.485095 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.485147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.485112 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.485147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.485126 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adb8cf5-05ef-49af-ac31-714008b9f57c-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.485147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.485141 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adb8cf5-05ef-49af-ac31-714008b9f57c-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.485588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.485558 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df4b69-5198-4a06-8df2-8ea612beca27-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.496686 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:18:47.496652 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:18:47.543431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.543402 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9f7\" (UniqueName: \"kubernetes.io/projected/33df4b69-5198-4a06-8df2-8ea612beca27-kube-api-access-sh9f7\") pod \"payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.593777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.593748 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:47.604936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.604910 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" podUID="33665aab-7efa-45e3-9e3c-f6dd9f4ef091" containerName="payload-processing" containerID="cri-o://9b2dfb98fb2675ea006893192c395b0931171ba08c54cd61ec6d49760d8ba8ea" gracePeriod=25 Oct 31 15:18:47.606714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.606693 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:47.713775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.713728 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" podUID="ac41e17e-b7f0-42c7-8900-08150a0365f5" containerName="payload-pre-processing" containerID="cri-o://0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76" gracePeriod=25 Oct 31 15:18:47.804591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.804456 4865 generic.go:356] "Generic (PLEG): container finished" podID="33665aab-7efa-45e3-9e3c-f6dd9f4ef091" containerID="9b2dfb98fb2675ea006893192c395b0931171ba08c54cd61ec6d49760d8ba8ea" exitCode=0 Oct 31 15:18:47.804591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.804522 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" event={"ID":"33665aab-7efa-45e3-9e3c-f6dd9f4ef091","Type":"ContainerDied","Data":"9b2dfb98fb2675ea006893192c395b0931171ba08c54cd61ec6d49760d8ba8ea"} Oct 31 15:18:47.807157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.807128 4865 generic.go:356] "Generic (PLEG): container finished" podID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerID="628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3" exitCode=0 Oct 31 15:18:47.807350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.807332 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" event={"ID":"2adb8cf5-05ef-49af-ac31-714008b9f57c","Type":"ContainerDied","Data":"628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3"} Oct 31 15:18:47.807437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.807380 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" event={"ID":"2adb8cf5-05ef-49af-ac31-714008b9f57c","Type":"ContainerDied","Data":"67d48a109449251ad14f2be3c6e18d4efcd33c266390ae37bbe8e94bccf1c811"} Oct 31 15:18:47.807437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.807410 4865 scope.go:122] "RemoveContainer" containerID="628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3" Oct 31 15:18:47.807652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.807600 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh" Oct 31 15:18:47.833689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.833659 4865 scope.go:122] "RemoveContainer" containerID="628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3" Oct 31 15:18:47.834076 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:18:47.834056 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3\": container with ID starting with 628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3 not found: ID does not exist" containerID="628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3" Oct 31 15:18:47.834157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.834084 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3"} err="failed to get container status \"628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3\": rpc error: code = NotFound desc = could not find container \"628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3\": container with ID starting with 628a9a52fe8dd26522664a91e7073bb64ae0dd051c73da91d8ac4eb5fed283e3 not found: ID does not exist" Oct 31 15:18:47.901989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.901914 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:18:47.974243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.974214 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq"] Oct 31 15:18:47.980424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.980386 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh"] Oct 31 15:18:47.991691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.991663 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-kube-api-access-rl2gc\" (UniqueName: \"kubernetes.io/projected/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-kube-api-access-rl2gc\") pod \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " Oct 31 15:18:47.991803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.991741 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-plugins-config-volume\") pod \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\" (UID: \"33665aab-7efa-45e3-9e3c-f6dd9f4ef091\") " Oct 31 15:18:47.992139 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.992115 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-plugins-config-volume" pod "33665aab-7efa-45e3-9e3c-f6dd9f4ef091" (UID: "33665aab-7efa-45e3-9e3c-f6dd9f4ef091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:47.992317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.992300 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.994672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:47.994651 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-kube-api-access-rl2gc" pod "33665aab-7efa-45e3-9e3c-f6dd9f4ef091" (UID: "33665aab-7efa-45e3-9e3c-f6dd9f4ef091"). InnerVolumeSpecName "kube-api-access-rl2gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:48.012029 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:18:48.012004 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33df4b69_5198_4a06_8df2_8ea612beca27.slice/crio-55048367b9f3d8d24df3da1894c9485918a176656b7a7c9942951d263e78816c WatchSource:0}: Error finding container 55048367b9f3d8d24df3da1894c9485918a176656b7a7c9942951d263e78816c: Status 404 returned error can't find the container with id 55048367b9f3d8d24df3da1894c9485918a176656b7a7c9942951d263e78816c Oct 31 15:18:48.025660 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:18:48.025635 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e233da_d5cd_41d0_8b04_af6edc905ca8.slice/crio-85782ee7a075731f5e43a44cd90c9ac35e6d06068daa7392b6ca025318ac4948 WatchSource:0}: Error finding container 85782ee7a075731f5e43a44cd90c9ac35e6d06068daa7392b6ca025318ac4948: Status 404 returned error can't find the container with id 85782ee7a075731f5e43a44cd90c9ac35e6d06068daa7392b6ca025318ac4948 Oct 31 15:18:48.029913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.029883 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-b6vkh"] Oct 31 15:18:48.030344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.030324 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:18:48.033196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.033172 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl"] Oct 31 15:18:48.093751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.093727 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2gc\" (UniqueName: \"kubernetes.io/projected/33665aab-7efa-45e3-9e3c-f6dd9f4ef091-kube-api-access-rl2gc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:48.195182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.195103 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac41e17e-b7f0-42c7-8900-08150a0365f5-kube-api-access-pdr6j\" (UniqueName: \"kubernetes.io/projected/ac41e17e-b7f0-42c7-8900-08150a0365f5-kube-api-access-pdr6j\") pod \"ac41e17e-b7f0-42c7-8900-08150a0365f5\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " Oct 31 15:18:48.195182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.195162 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac41e17e-b7f0-42c7-8900-08150a0365f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac41e17e-b7f0-42c7-8900-08150a0365f5-plugins-config-volume\") pod \"ac41e17e-b7f0-42c7-8900-08150a0365f5\" (UID: \"ac41e17e-b7f0-42c7-8900-08150a0365f5\") " Oct 31 15:18:48.195571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.195548 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac41e17e-b7f0-42c7-8900-08150a0365f5-plugins-config-volume" pod "ac41e17e-b7f0-42c7-8900-08150a0365f5" (UID: "ac41e17e-b7f0-42c7-8900-08150a0365f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:48.197410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.197388 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac41e17e-b7f0-42c7-8900-08150a0365f5-kube-api-access-pdr6j" pod "ac41e17e-b7f0-42c7-8900-08150a0365f5" (UID: "ac41e17e-b7f0-42c7-8900-08150a0365f5"). InnerVolumeSpecName "kube-api-access-pdr6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:48.297033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.297004 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdr6j\" (UniqueName: \"kubernetes.io/projected/ac41e17e-b7f0-42c7-8900-08150a0365f5-kube-api-access-pdr6j\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:48.297033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.297033 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac41e17e-b7f0-42c7-8900-08150a0365f5-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:48.498600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.498568 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" path="/var/lib/kubelet/pods/2adb8cf5-05ef-49af-ac31-714008b9f57c/volumes" Oct 31 15:18:48.815237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.815140 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" event={"ID":"33665aab-7efa-45e3-9e3c-f6dd9f4ef091","Type":"ContainerDied","Data":"aa63787fa7cfcb7dbd019269996c9677fac51cd83ade4ec62cebe4910c767ae1"} Oct 31 15:18:48.815237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.815150 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw" Oct 31 15:18:48.815784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.815240 4865 scope.go:122] "RemoveContainer" containerID="9b2dfb98fb2675ea006893192c395b0931171ba08c54cd61ec6d49760d8ba8ea" Oct 31 15:18:48.819052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.819026 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" event={"ID":"d6e233da-d5cd-41d0-8b04-af6edc905ca8","Type":"ContainerStarted","Data":"8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903"} Oct 31 15:18:48.819180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.819077 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" event={"ID":"d6e233da-d5cd-41d0-8b04-af6edc905ca8","Type":"ContainerStarted","Data":"85782ee7a075731f5e43a44cd90c9ac35e6d06068daa7392b6ca025318ac4948"} Oct 31 15:18:48.819180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.819108 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:18:48.820454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.820428 4865 generic.go:356] "Generic (PLEG): container finished" podID="ac41e17e-b7f0-42c7-8900-08150a0365f5" containerID="0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76" exitCode=0 Oct 31 15:18:48.820577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.820495 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" event={"ID":"ac41e17e-b7f0-42c7-8900-08150a0365f5","Type":"ContainerDied","Data":"0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76"} Oct 31 15:18:48.820577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.820534 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" event={"ID":"ac41e17e-b7f0-42c7-8900-08150a0365f5","Type":"ContainerDied","Data":"49e227634f97bc6e594f1b45f3ec8b50c5928927d36214cd117d273f3ce30223"} Oct 31 15:18:48.820577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.820500 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl" Oct 31 15:18:48.822170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.822147 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" event={"ID":"33df4b69-5198-4a06-8df2-8ea612beca27","Type":"ContainerStarted","Data":"9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28"} Oct 31 15:18:48.822266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.822200 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" event={"ID":"33df4b69-5198-4a06-8df2-8ea612beca27","Type":"ContainerStarted","Data":"55048367b9f3d8d24df3da1894c9485918a176656b7a7c9942951d263e78816c"} Oct 31 15:18:48.822356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.822335 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:48.826793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.826777 4865 scope.go:122] "RemoveContainer" containerID="0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76" Oct 31 15:18:48.836316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.836298 4865 scope.go:122] "RemoveContainer" containerID="0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76" Oct 31 15:18:48.836622 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:18:48.836598 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76\": container with ID starting with 0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76 not found: ID does not exist" containerID="0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76" Oct 31 15:18:48.836701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.836632 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76"} err="failed to get container status \"0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76\": rpc error: code = NotFound desc = could not find container \"0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76\": container with ID starting with 0649a7c71eb7891c46485e93195611eb8656ab14736290ed12222c352e9f4f76 not found: ID does not exist" Oct 31 15:18:48.987066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:48.987009 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" podStartSLOduration=2.986995918 podStartE2EDuration="2.986995918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:18:48.983735388 +0000 UTC m=+1974.603286800" watchObservedRunningTime="2026-10-31 15:18:48.986995918 +0000 UTC m=+1974.606547354" Oct 31 15:18:49.127927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:49.125563 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl"] Oct 31 15:18:49.163589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:49.163550 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-f54d5cb5f-92thl"] Oct 31 15:18:49.223506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:49.223459 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw"] Oct 31 15:18:49.226327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:49.226305 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-6d978786bf-pqmhw"] Oct 31 15:18:49.338110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:49.338054 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" podStartSLOduration=2.338038912 podStartE2EDuration="2.338038912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:18:49.335965538 +0000 UTC m=+1974.955516962" watchObservedRunningTime="2026-10-31 15:18:49.338038912 +0000 UTC m=+1974.957590322" Oct 31 15:18:50.498659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:50.498619 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33665aab-7efa-45e3-9e3c-f6dd9f4ef091" path="/var/lib/kubelet/pods/33665aab-7efa-45e3-9e3c-f6dd9f4ef091/volumes" Oct 31 15:18:50.499205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:50.499073 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac41e17e-b7f0-42c7-8900-08150a0365f5" path="/var/lib/kubelet/pods/ac41e17e-b7f0-42c7-8900-08150a0365f5/volumes" Oct 31 15:18:51.313108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:51.313056 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:51.448360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:51.448330 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:51.702863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:51.702792 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:52.648874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:52.648838 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:55.921992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:55.921957 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:56.372450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:56.372418 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:57.323988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:57.323958 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:58.416023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:58.415979 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:18:59.835310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:59.835274 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:18:59.835731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:18:59.835350 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:19:01.233529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:01.233492 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:01.385731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:01.385694 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:01.509178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:01.509086 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:06.208550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:06.208517 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:10.909984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:10.909950 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:15.688089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:15.688056 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:15.933663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:15.933634 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:19:17.492365 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:17.492321 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:19:17.616331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:17.616297 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:18.932893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:18.932685 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:20.816036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:20.815995 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:23.692647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:23.692609 4865 patch_prober.go:29] interesting pod/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:19:23.693064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:23.692706 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:19:24.800516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:24.800421 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:25.109130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.109047 4865 generic.go:356] "Generic (PLEG): container finished" podID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerID="54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f" exitCode=0 Oct 31 15:19:25.109286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.109136 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" event={"ID":"5d091acc-c288-43e4-99dd-a505a4f62fb0","Type":"ContainerDied","Data":"54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f"} Oct 31 15:19:25.109582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.109568 4865 scope.go:122] "RemoveContainer" containerID="54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f" Oct 31 15:19:25.111169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.111138 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:19:25.111274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.111209 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:19:25.111274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.111243 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:19:25.505250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.505211 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:25.543396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.543363 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw"] Oct 31 15:19:25.803553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.794082 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:25.952141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:25.952109 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:26.131148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:26.131062 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" event={"ID":"5d091acc-c288-43e4-99dd-a505a4f62fb0","Type":"ContainerStarted","Data":"ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f"} Oct 31 15:19:26.219239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:26.218835 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:19:26.219239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:26.218957 4865 patch_prober.go:29] interesting pod/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 15:19:26.219239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:26.218997 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 15:19:27.200873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:27.200830 4865 patch_prober.go:29] interesting pod/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 15:19:27.204667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:27.201427 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 15:19:27.258692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:27.258660 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:19:27.258956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:27.258923 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:19:28.167743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.167708 4865 generic.go:356] "Generic (PLEG): container finished" podID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerID="da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524" exitCode=0 Oct 31 15:19:28.167943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.167781 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" event={"ID":"e15e1799-085c-409b-8365-6f8ea4cdeda1","Type":"ContainerDied","Data":"da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524"} Oct 31 15:19:28.168896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.168869 4865 scope.go:122] "RemoveContainer" containerID="da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524" Oct 31 15:19:28.175144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.175087 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:19:28.175278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.175253 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:19:28.175369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.175349 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:19:28.210505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.208777 4865 patch_prober.go:29] interesting pod/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 15:19:28.210505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:28.208836 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 15:19:29.177246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:29.177213 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" event={"ID":"e15e1799-085c-409b-8365-6f8ea4cdeda1","Type":"ContainerStarted","Data":"c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0"} Oct 31 15:19:29.199883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:29.199846 4865 patch_prober.go:29] interesting pod/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 15:19:29.200180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:29.200151 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 15:19:29.867396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:29.867366 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:19:29.867911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:29.867503 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 15:19:29.867911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:29.867677 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 15:19:30.201925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:30.201829 4865 patch_prober.go:29] interesting pod/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 15:19:30.202204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:30.202172 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 15:19:30.420496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:30.419361 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:30.545403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:30.545350 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" podUID="6e31af48-0ef0-40dc-97c3-473e46736456" containerName="payload-pre-processing" containerID="cri-o://04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6" gracePeriod=25 Oct 31 15:19:30.612097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:30.612063 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:30.868027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:30.867589 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 15:19:30.868027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:30.867661 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 15:19:31.017495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.016709 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:19:31.072978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.072946 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e31af48-0ef0-40dc-97c3-473e46736456-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e31af48-0ef0-40dc-97c3-473e46736456-plugins-config-volume\") pod \"6e31af48-0ef0-40dc-97c3-473e46736456\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " Oct 31 15:19:31.073355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.073328 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e31af48-0ef0-40dc-97c3-473e46736456-plugins-config-volume" pod "6e31af48-0ef0-40dc-97c3-473e46736456" (UID: "6e31af48-0ef0-40dc-97c3-473e46736456"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:31.073508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.073481 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e31af48-0ef0-40dc-97c3-473e46736456-kube-api-access-k9fsk\" (UniqueName: \"kubernetes.io/projected/6e31af48-0ef0-40dc-97c3-473e46736456-kube-api-access-k9fsk\") pod \"6e31af48-0ef0-40dc-97c3-473e46736456\" (UID: \"6e31af48-0ef0-40dc-97c3-473e46736456\") " Oct 31 15:19:31.074021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.074007 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e31af48-0ef0-40dc-97c3-473e46736456-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:31.075715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.075690 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e31af48-0ef0-40dc-97c3-473e46736456-kube-api-access-k9fsk" pod "6e31af48-0ef0-40dc-97c3-473e46736456" (UID: "6e31af48-0ef0-40dc-97c3-473e46736456"). InnerVolumeSpecName "kube-api-access-k9fsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:31.175814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.175733 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fsk\" (UniqueName: \"kubernetes.io/projected/6e31af48-0ef0-40dc-97c3-473e46736456-kube-api-access-k9fsk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:31.251386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.247332 4865 patch_prober.go:29] interesting pod/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 15:19:31.251386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.247397 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 15:19:31.253594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.253547 4865 generic.go:356] "Generic (PLEG): container finished" podID="6e31af48-0ef0-40dc-97c3-473e46736456" containerID="04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6" exitCode=0 Oct 31 15:19:31.253594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.253594 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" Oct 31 15:19:31.253802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.253778 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" event={"ID":"6e31af48-0ef0-40dc-97c3-473e46736456","Type":"ContainerDied","Data":"04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6"} Oct 31 15:19:31.253907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.253893 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw" event={"ID":"6e31af48-0ef0-40dc-97c3-473e46736456","Type":"ContainerDied","Data":"436ed4defcc550cdb20b0be235fb2d06b27861b6eb8eab60cf153bd44c12a102"} Oct 31 15:19:31.254017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.253969 4865 scope.go:122] "RemoveContainer" containerID="04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6" Oct 31 15:19:31.267308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.267209 4865 scope.go:122] "RemoveContainer" containerID="04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6" Oct 31 15:19:31.267564 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:31.267540 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6\": container with ID starting with 04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6 not found: ID does not exist" containerID="04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6" Oct 31 15:19:31.267638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.267575 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6"} err="failed to get container status \"04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6\": rpc error: code = NotFound desc = could not find container \"04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6\": container with ID starting with 04eab791ce6d3e87e3c31e96fb6df8e6c359dcd0d368d8cacea43de885b279e6 not found: ID does not exist" Oct 31 15:19:31.427239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.427144 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw"] Oct 31 15:19:31.430160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.430132 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f76cce-68d9d8f79-p8rmw"] Oct 31 15:19:31.867271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.867242 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 15:19:31.867453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:31.867295 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 15:19:32.220486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:32.220425 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:19:32.221026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:32.221005 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:19:32.221872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:32.221849 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:19:32.526888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:32.526805 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e31af48-0ef0-40dc-97c3-473e46736456" path="/var/lib/kubelet/pods/6e31af48-0ef0-40dc-97c3-473e46736456/volumes" Oct 31 15:19:32.866904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:32.866825 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 15:19:32.866904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:32.866876 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 15:19:33.866757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:33.866725 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 15:19:33.867168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:33.866775 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 15:19:34.871043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:34.871016 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:19:34.871490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:34.871459 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:19:34.872176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:34.872157 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:19:36.900844 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:36.900759 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:41.784608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:41.784569 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:42.065891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:42.065820 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj"] Oct 31 15:19:42.066038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:42.066018 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" containerID="cri-o://c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5" gracePeriod=30 Oct 31 15:19:44.128643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:44.128607 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd"] Oct 31 15:19:44.129148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:44.128855 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" containerID="cri-o://08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522" gracePeriod=30 Oct 31 15:19:44.873449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:44.873406 4865 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:19:44.873639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:44.873463 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:19:45.257415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:45.257381 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:45.740741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:45.740704 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:46.846206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.846167 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht"] Oct 31 15:19:46.846990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.846967 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" Oct 31 15:19:46.847094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.846991 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" Oct 31 15:19:46.847094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847019 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ac41e17e-b7f0-42c7-8900-08150a0365f5" containerName="payload-pre-processing" Oct 31 15:19:46.847094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847026 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac41e17e-b7f0-42c7-8900-08150a0365f5" containerName="payload-pre-processing" Oct 31 15:19:46.847094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847046 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6e31af48-0ef0-40dc-97c3-473e46736456" containerName="payload-pre-processing" Oct 31 15:19:46.847094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847053 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e31af48-0ef0-40dc-97c3-473e46736456" containerName="payload-pre-processing" Oct 31 15:19:46.847358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847096 4865 cpu_manager.go:414] "containerMap: removing container" podUID="33665aab-7efa-45e3-9e3c-f6dd9f4ef091" containerName="payload-processing" Oct 31 15:19:46.847358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847104 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33665aab-7efa-45e3-9e3c-f6dd9f4ef091" containerName="payload-processing" Oct 31 15:19:46.847358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847253 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="33665aab-7efa-45e3-9e3c-f6dd9f4ef091" containerName="payload-processing" Oct 31 15:19:46.847358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847275 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2adb8cf5-05ef-49af-ac31-714008b9f57c" containerName="istio-proxy" Oct 31 15:19:46.847358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847287 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac41e17e-b7f0-42c7-8900-08150a0365f5" containerName="payload-pre-processing" Oct 31 15:19:46.847358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.847298 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e31af48-0ef0-40dc-97c3-473e46736456" containerName="payload-pre-processing" Oct 31 15:19:46.852268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.852246 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:46.907680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.907655 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht"] Oct 31 15:19:46.954391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.954365 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad5bfe-dockercfg-cjlpd\"" Oct 31 15:19:46.954579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:46.954428 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad5bfe\"" Oct 31 15:19:47.014593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.014563 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhmk\" (UniqueName: \"kubernetes.io/projected/ce936a25-6ca2-420e-9447-71682a664773-kube-api-access-hrhmk\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:47.014953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.014918 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce936a25-6ca2-420e-9447-71682a664773-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:47.115970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.115939 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhmk\" (UniqueName: \"kubernetes.io/projected/ce936a25-6ca2-420e-9447-71682a664773-kube-api-access-hrhmk\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:47.116141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.116125 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce936a25-6ca2-420e-9447-71682a664773-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:47.117114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.117089 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce936a25-6ca2-420e-9447-71682a664773-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:47.174654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.174624 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhmk\" (UniqueName: \"kubernetes.io/projected/ce936a25-6ca2-420e-9447-71682a664773-kube-api-access-hrhmk\") pod \"payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:47.189905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.189881 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w"] Oct 31 15:19:47.190087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.190062 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" containerID="cri-o://341f3924b2653e5780a287b6c991aa1bd74ff5bd4e89c8aa68fed365bab84e08" gracePeriod=30 Oct 31 15:19:47.346445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.346416 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:47.376945 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.376920 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:19:47.386997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.386968 4865 generic.go:356] "Generic (PLEG): container finished" podID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerID="c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5" exitCode=0 Oct 31 15:19:47.387150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.387016 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" Oct 31 15:19:47.387150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.387050 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" event={"ID":"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00","Type":"ContainerDied","Data":"c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5"} Oct 31 15:19:47.387150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.387114 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj" event={"ID":"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00","Type":"ContainerDied","Data":"11bf1bf65b03cf58e3546dbfb5fe217bd7b5f5ab38689505ee0961df9ef4a2d6"} Oct 31 15:19:47.387150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.387128 4865 scope.go:122] "RemoveContainer" containerID="c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5" Oct 31 15:19:47.398933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.398911 4865 scope.go:122] "RemoveContainer" containerID="c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5" Oct 31 15:19:47.399223 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:47.399203 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5\": container with ID starting with c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5 not found: ID does not exist" containerID="c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5" Oct 31 15:19:47.399307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.399232 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5"} err="failed to get container status \"c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5\": rpc error: code = NotFound desc = could not find container \"c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5\": container with ID starting with c2f6a201a14e1ba1cf8d13eab40c20723192b9911e01f62a3a6ab1351b3fe6f5 not found: ID does not exist" Oct 31 15:19:47.466922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.466897 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:47.500216 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:47.500179 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:19:47.521790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.521748 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istiod-ca-cert\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.521982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.521910 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-kube-api-access-7nk4l\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-kube-api-access-7nk4l\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.522048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.522005 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-data\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.522114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.522063 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-envoy\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.522173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.522144 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-certs\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.522245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.522178 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-token\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-token\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.522245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.522211 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-podinfo\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.522367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.522276 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-socket\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.522367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.522334 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-credential-socket\") pod \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\" (UID: \"bbab9b7d-8c9d-41aa-8016-61e4a20cfc00\") " Oct 31 15:19:47.523194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.523167 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-credential-socket" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:47.523567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.523538 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istiod-ca-cert" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:47.523997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.523971 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-certs" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:47.524350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.524325 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-socket" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:47.524442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.524421 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-data" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:47.526515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.526447 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-token" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:47.526666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.526646 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-podinfo" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:19:47.526777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.526755 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-envoy" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:47.526842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.526782 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-kube-api-access-7nk4l" pod "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" (UID: "bbab9b7d-8c9d-41aa-8016-61e4a20cfc00"). InnerVolumeSpecName "kube-api-access-7nk4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:47.625136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625037 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nk4l\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-kube-api-access-7nk4l\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625082 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625098 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625112 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625126 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625140 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625154 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625168 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.625645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.625180 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:47.654185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.654155 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm"] Oct 31 15:19:47.654828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.654813 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" Oct 31 15:19:47.654881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.654830 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" Oct 31 15:19:47.654939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.654930 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" containerName="istio-proxy" Oct 31 15:19:47.659524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.659505 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:47.714950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.714923 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm"] Oct 31 15:19:47.827234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.827201 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6p7w\" (UniqueName: \"kubernetes.io/projected/26cde22a-296d-4ee4-9c93-d4e4a25aec90-kube-api-access-q6p7w\") pod \"payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:47.827395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.827241 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cde22a-296d-4ee4-9c93-d4e4a25aec90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:47.863733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.863708 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht"] Oct 31 15:19:47.864153 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:19:47.863843 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce936a25_6ca2_420e_9447_71682a664773.slice/crio-9d600c91386563d44960abb6688af2492ca4c7ed811549a71b8086f000949f02 WatchSource:0}: Error finding container 9d600c91386563d44960abb6688af2492ca4c7ed811549a71b8086f000949f02: Status 404 returned error can't find the container with id 9d600c91386563d44960abb6688af2492ca4c7ed811549a71b8086f000949f02 Oct 31 15:19:47.890498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.890454 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj"] Oct 31 15:19:47.893382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.893360 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-w9scj"] Oct 31 15:19:47.928948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.928918 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6p7w\" (UniqueName: \"kubernetes.io/projected/26cde22a-296d-4ee4-9c93-d4e4a25aec90-kube-api-access-q6p7w\") pod \"payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:47.929049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.928969 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cde22a-296d-4ee4-9c93-d4e4a25aec90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:47.929705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.929676 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cde22a-296d-4ee4-9c93-d4e4a25aec90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:47.986962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:47.986934 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6p7w\" (UniqueName: \"kubernetes.io/projected/26cde22a-296d-4ee4-9c93-d4e4a25aec90-kube-api-access-q6p7w\") pod \"payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:48.275413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.275374 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:48.307016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.306984 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:48.420929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.420885 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" event={"ID":"ce936a25-6ca2-420e-9447-71682a664773","Type":"ContainerStarted","Data":"5cbff1fb8acd20c19613a0f666f14376f549b5d571db3d2fa7b47d1c58ed12be"} Oct 31 15:19:48.421166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.420991 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" event={"ID":"ce936a25-6ca2-420e-9447-71682a664773","Type":"ContainerStarted","Data":"9d600c91386563d44960abb6688af2492ca4c7ed811549a71b8086f000949f02"} Oct 31 15:19:48.422121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.422099 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:19:48.497483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.497443 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbab9b7d-8c9d-41aa-8016-61e4a20cfc00" path="/var/lib/kubelet/pods/bbab9b7d-8c9d-41aa-8016-61e4a20cfc00/volumes" Oct 31 15:19:48.539622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.539526 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" podStartSLOduration=2.539513044 podStartE2EDuration="2.539513044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:19:48.537791379 +0000 UTC m=+2034.157342791" watchObservedRunningTime="2026-10-31 15:19:48.539513044 +0000 UTC m=+2034.159064454" Oct 31 15:19:48.656384 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:19:48.656357 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cde22a_296d_4ee4_9c93_d4e4a25aec90.slice/crio-680a240df23ff367fc32839c786565524864d49b14abe3d56e41ca296159f0e0 WatchSource:0}: Error finding container 680a240df23ff367fc32839c786565524864d49b14abe3d56e41ca296159f0e0: Status 404 returned error can't find the container with id 680a240df23ff367fc32839c786565524864d49b14abe3d56e41ca296159f0e0 Oct 31 15:19:48.657056 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:48.657034 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm"] Oct 31 15:19:49.407675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.407613 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:19:49.430888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.430854 4865 generic.go:356] "Generic (PLEG): container finished" podID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerID="08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522" exitCode=0 Oct 31 15:19:49.431052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.430898 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" Oct 31 15:19:49.431052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.430934 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" event={"ID":"9113d241-1dd7-4d9f-a2af-c8043456e64b","Type":"ContainerDied","Data":"08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522"} Oct 31 15:19:49.431052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.430990 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd" event={"ID":"9113d241-1dd7-4d9f-a2af-c8043456e64b","Type":"ContainerDied","Data":"40f9f8d6fb503c8b78ac0e8aab1caa8fed3a2ab84b61090801119876c5d32df2"} Oct 31 15:19:49.431052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.431018 4865 scope.go:122] "RemoveContainer" containerID="08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522" Oct 31 15:19:49.432999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.432976 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" event={"ID":"26cde22a-296d-4ee4-9c93-d4e4a25aec90","Type":"ContainerStarted","Data":"06d2e1c75f5ca973658e1a88d4530517397e5502170b699e418c536f88feb630"} Oct 31 15:19:49.433119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.433022 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" event={"ID":"26cde22a-296d-4ee4-9c93-d4e4a25aec90","Type":"ContainerStarted","Data":"680a240df23ff367fc32839c786565524864d49b14abe3d56e41ca296159f0e0"} Oct 31 15:19:49.433119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.433051 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:19:49.442950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.442931 4865 scope.go:122] "RemoveContainer" containerID="08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522" Oct 31 15:19:49.443228 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:49.443203 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522\": container with ID starting with 08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522 not found: ID does not exist" containerID="08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522" Oct 31 15:19:49.443289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.443236 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522"} err="failed to get container status \"08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522\": rpc error: code = NotFound desc = could not find container \"08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522\": container with ID starting with 08b81655a4b01b8cc17be0c4b726dc2113af1c75a90dd9a01823b57ef7a69522 not found: ID does not exist" Oct 31 15:19:49.554152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554124 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-data\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554161 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-credential-socket\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554220 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-socket\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554261 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-envoy\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554293 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9113d241-1dd7-4d9f-a2af-c8043456e64b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9113d241-1dd7-4d9f-a2af-c8043456e64b-istiod-ca-cert\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554411 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-certs\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554441 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-data" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:49.554760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554582 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-credential-socket" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:49.554760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554618 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-podinfo\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554631 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-socket" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:49.554760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554683 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-kube-api-access-8z9jb\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-kube-api-access-8z9jb\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.554760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554710 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-token\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-token\") pod \"9113d241-1dd7-4d9f-a2af-c8043456e64b\" (UID: \"9113d241-1dd7-4d9f-a2af-c8043456e64b\") " Oct 31 15:19:49.555215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554799 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-certs" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:49.555215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.554892 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9113d241-1dd7-4d9f-a2af-c8043456e64b-istiod-ca-cert" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:49.555643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.555509 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.555643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.555540 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.555643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.555557 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.555643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.555571 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9113d241-1dd7-4d9f-a2af-c8043456e64b-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.555643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.555586 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.557527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.557506 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-envoy" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:49.557751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.557726 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-podinfo" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:19:49.557829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.557734 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-kube-api-access-8z9jb" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "kube-api-access-8z9jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:49.558020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.558002 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-token" pod "9113d241-1dd7-4d9f-a2af-c8043456e64b" (UID: "9113d241-1dd7-4d9f-a2af-c8043456e64b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:49.641912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.641851 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" podStartSLOduration=2.641835271 podStartE2EDuration="2.641835271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:19:49.640291649 +0000 UTC m=+2035.259843061" watchObservedRunningTime="2026-10-31 15:19:49.641835271 +0000 UTC m=+2035.261386708" Oct 31 15:19:49.656613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.656588 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9jb\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-kube-api-access-8z9jb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.656613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.656612 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.656812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.656621 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.656812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.656631 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9113d241-1dd7-4d9f-a2af-c8043456e64b-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:49.855279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.855247 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd"] Oct 31 15:19:49.861087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:49.861062 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-4rwdd"] Oct 31 15:19:50.509676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:50.509642 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" path="/var/lib/kubelet/pods/9113d241-1dd7-4d9f-a2af-c8043456e64b/volumes" Oct 31 15:19:51.389724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:51.389684 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9cm2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:19:51.389934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:51.389761 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:19:51.957308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:51.957277 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk"] Oct 31 15:19:51.957729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:51.957481 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" containerID="cri-o://ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24" gracePeriod=30 Oct 31 15:19:52.458714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.458685 4865 generic.go:356] "Generic (PLEG): container finished" podID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerID="341f3924b2653e5780a287b6c991aa1bd74ff5bd4e89c8aa68fed365bab84e08" exitCode=0 Oct 31 15:19:52.458881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.458767 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" event={"ID":"663838ff-9d0b-49d3-8cd4-62bf619754c8","Type":"ContainerDied","Data":"341f3924b2653e5780a287b6c991aa1bd74ff5bd4e89c8aa68fed365bab84e08"} Oct 31 15:19:52.458881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.458833 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" event={"ID":"663838ff-9d0b-49d3-8cd4-62bf619754c8","Type":"ContainerDied","Data":"46b5297380144715c0001bb828d6c54f213a38c639cf1da7f226c2f41543d01f"} Oct 31 15:19:52.458881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.458857 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46b5297380144715c0001bb828d6c54f213a38c639cf1da7f226c2f41543d01f" Oct 31 15:19:52.473440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.473382 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:19:52.493085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493065 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-podinfo\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.493208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493125 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-envoy\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.493208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493151 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-data\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.493327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493230 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-credential-socket\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.493327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493263 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-kube-api-access-sh8k7\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-kube-api-access-sh8k7\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.493442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493327 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-token\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-token\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.493774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493412 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-certs\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.493936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493519 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-data" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.494107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493597 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-credential-socket" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.494107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.493702 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-certs" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.494107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.494044 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663838ff-9d0b-49d3-8cd4-62bf619754c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663838ff-9d0b-49d3-8cd4-62bf619754c8-istiod-ca-cert\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.494107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.494084 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-socket\") pod \"663838ff-9d0b-49d3-8cd4-62bf619754c8\" (UID: \"663838ff-9d0b-49d3-8cd4-62bf619754c8\") " Oct 31 15:19:52.494645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.494624 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.494762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.494649 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.494762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.494664 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.495176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.495109 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-socket" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.495397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.495363 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663838ff-9d0b-49d3-8cd4-62bf619754c8-istiod-ca-cert" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:52.496356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.496320 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-envoy" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.496356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.496340 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-podinfo" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:19:52.496521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.496368 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-kube-api-access-sh8k7" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "kube-api-access-sh8k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:52.496521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.496395 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-token" pod "663838ff-9d0b-49d3-8cd4-62bf619754c8" (UID: "663838ff-9d0b-49d3-8cd4-62bf619754c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:52.596344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.596310 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh8k7\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-kube-api-access-sh8k7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.596344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.596346 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.596575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.596359 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663838ff-9d0b-49d3-8cd4-62bf619754c8-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.596575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.596373 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.596575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.596387 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.596575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.596400 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663838ff-9d0b-49d3-8cd4-62bf619754c8-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.670186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.670149 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:19:52.670402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.670224 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:19:52.946865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:52.946786 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:53.097692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:53.097656 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:53.226451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:53.226414 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:53.465530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:53.465498 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w" Oct 31 15:19:53.633565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:53.633481 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w"] Oct 31 15:19:53.684358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:53.684316 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9cm2w"] Oct 31 15:19:54.497577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:54.497547 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" path="/var/lib/kubelet/pods/663838ff-9d0b-49d3-8cd4-62bf619754c8/volumes" Oct 31 15:19:56.271596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:56.271570 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:56.780490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:56.780441 4865 scope.go:122] "RemoveContainer" containerID="341f3924b2653e5780a287b6c991aa1bd74ff5bd4e89c8aa68fed365bab84e08" Oct 31 15:19:57.223827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.223795 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:57.251775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.251745 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:19:57.367771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.367690 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-data" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370388 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-data\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370502 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-certs\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370608 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-podinfo\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370713 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-socket\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370762 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istiod-ca-cert\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370793 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-token\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-token\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370828 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-credential-socket\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370865 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-envoy\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.370971 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-kube-api-access-7rqw4\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-kube-api-access-7rqw4\") pod \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\" (UID: \"0e312e76-afbd-49b9-82bf-c7a1d286bde9\") " Oct 31 15:19:57.372054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.371587 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.388305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.382952 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-certs" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:57.388305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.384800 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istiod-ca-cert" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:57.388305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.385385 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-socket" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:57.388305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.385561 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-credential-socket" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:57.388711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.388686 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-token" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:57.389262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.389219 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-kube-api-access-7rqw4" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "kube-api-access-7rqw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:57.391234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.391208 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-envoy" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:57.392858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.392820 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-podinfo" pod "0e312e76-afbd-49b9-82bf-c7a1d286bde9" (UID: "0e312e76-afbd-49b9-82bf-c7a1d286bde9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:19:57.473279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473234 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.473279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473276 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.473593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473292 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.473593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473308 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.473593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473321 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.473593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473333 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.473593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473349 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e312e76-afbd-49b9-82bf-c7a1d286bde9-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.473593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.473362 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqw4\" (UniqueName: \"kubernetes.io/projected/0e312e76-afbd-49b9-82bf-c7a1d286bde9-kube-api-access-7rqw4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.495715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.495682 4865 generic.go:356] "Generic (PLEG): container finished" podID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerID="ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24" exitCode=0 Oct 31 15:19:57.495874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.495739 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" Oct 31 15:19:57.495874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.495760 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" event={"ID":"0e312e76-afbd-49b9-82bf-c7a1d286bde9","Type":"ContainerDied","Data":"ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24"} Oct 31 15:19:57.495874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.495813 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk" event={"ID":"0e312e76-afbd-49b9-82bf-c7a1d286bde9","Type":"ContainerDied","Data":"c65449a2a1dd8449a9d760f10c7bda8e5777f207a8dc448f4d83bff5e8399a4c"} Oct 31 15:19:57.495874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.495838 4865 scope.go:122] "RemoveContainer" containerID="ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24" Oct 31 15:19:57.507808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.506569 4865 scope.go:122] "RemoveContainer" containerID="ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24" Oct 31 15:19:57.509201 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:57.508812 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24\": container with ID starting with ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24 not found: ID does not exist" containerID="ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24" Oct 31 15:19:57.509201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.508864 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24"} err="failed to get container status \"ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24\": rpc error: code = NotFound desc = could not find container \"ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24\": container with ID starting with ccd659a3e5823a60e321c8d2dd6200b7475270a18e8cdf92719443b87b44ad24 not found: ID does not exist" Oct 31 15:19:57.581448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.581417 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq"] Oct 31 15:19:57.581671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.581652 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" podUID="6230a2fd-0f3d-486c-b501-7228db16112e" containerName="payload-processing" containerID="cri-o://690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b" gracePeriod=30 Oct 31 15:19:57.621416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.621351 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" podUID="6230a2fd-0f3d-486c-b501-7228db16112e" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.180:9004: connect: connection refused" Oct 31 15:19:57.636936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.636909 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg"] Oct 31 15:19:57.637146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.637122 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" podUID="81a79b66-503f-454d-b279-4be18a34ad2d" containerName="payload-pre-processing" containerID="cri-o://6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05" gracePeriod=30 Oct 31 15:19:57.664144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.664070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:57.664345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.664324 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk"] Oct 31 15:19:57.670026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.669988 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-m62xk"] Oct 31 15:19:57.798612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.798579 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:57.899838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.899806 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:19:57.933146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.933123 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:19:57.990576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.990552 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6230a2fd-0f3d-486c-b501-7228db16112e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6230a2fd-0f3d-486c-b501-7228db16112e-plugins-config-volume\") pod \"6230a2fd-0f3d-486c-b501-7228db16112e\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " Oct 31 15:19:57.990700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.990654 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6230a2fd-0f3d-486c-b501-7228db16112e-kube-api-access-4q24g\" (UniqueName: \"kubernetes.io/projected/6230a2fd-0f3d-486c-b501-7228db16112e-kube-api-access-4q24g\") pod \"6230a2fd-0f3d-486c-b501-7228db16112e\" (UID: \"6230a2fd-0f3d-486c-b501-7228db16112e\") " Oct 31 15:19:57.990921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.990902 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6230a2fd-0f3d-486c-b501-7228db16112e-plugins-config-volume" pod "6230a2fd-0f3d-486c-b501-7228db16112e" (UID: "6230a2fd-0f3d-486c-b501-7228db16112e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:57.991074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.991060 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6230a2fd-0f3d-486c-b501-7228db16112e-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.992843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:57.992826 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6230a2fd-0f3d-486c-b501-7228db16112e-kube-api-access-4q24g" pod "6230a2fd-0f3d-486c-b501-7228db16112e" (UID: "6230a2fd-0f3d-486c-b501-7228db16112e"). InnerVolumeSpecName "kube-api-access-4q24g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:58.097499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.096053 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a79b66-503f-454d-b279-4be18a34ad2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a79b66-503f-454d-b279-4be18a34ad2d-plugins-config-volume\") pod \"81a79b66-503f-454d-b279-4be18a34ad2d\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " Oct 31 15:19:58.097499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.096232 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a79b66-503f-454d-b279-4be18a34ad2d-kube-api-access-26qcg\" (UniqueName: \"kubernetes.io/projected/81a79b66-503f-454d-b279-4be18a34ad2d-kube-api-access-26qcg\") pod \"81a79b66-503f-454d-b279-4be18a34ad2d\" (UID: \"81a79b66-503f-454d-b279-4be18a34ad2d\") " Oct 31 15:19:58.097499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.096900 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q24g\" (UniqueName: \"kubernetes.io/projected/6230a2fd-0f3d-486c-b501-7228db16112e-kube-api-access-4q24g\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:58.132520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.130617 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a79b66-503f-454d-b279-4be18a34ad2d-plugins-config-volume" pod "81a79b66-503f-454d-b279-4be18a34ad2d" (UID: "81a79b66-503f-454d-b279-4be18a34ad2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:58.147805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.147604 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a79b66-503f-454d-b279-4be18a34ad2d-kube-api-access-26qcg" pod "81a79b66-503f-454d-b279-4be18a34ad2d" (UID: "81a79b66-503f-454d-b279-4be18a34ad2d"). InnerVolumeSpecName "kube-api-access-26qcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:58.199983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.199950 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a79b66-503f-454d-b279-4be18a34ad2d-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:58.199983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.199986 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26qcg\" (UniqueName: \"kubernetes.io/projected/81a79b66-503f-454d-b279-4be18a34ad2d-kube-api-access-26qcg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:58.399735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.399629 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:19:58.496848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.496814 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" path="/var/lib/kubelet/pods/0e312e76-afbd-49b9-82bf-c7a1d286bde9/volumes" Oct 31 15:19:58.503829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.503795 4865 generic.go:356] "Generic (PLEG): container finished" podID="6230a2fd-0f3d-486c-b501-7228db16112e" containerID="690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b" exitCode=0 Oct 31 15:19:58.503999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.503848 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" Oct 31 15:19:58.503999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.503865 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" event={"ID":"6230a2fd-0f3d-486c-b501-7228db16112e","Type":"ContainerDied","Data":"690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b"} Oct 31 15:19:58.503999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.503916 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq" event={"ID":"6230a2fd-0f3d-486c-b501-7228db16112e","Type":"ContainerDied","Data":"fc9c2caf166944827093e16a2250ebaad5d0bc90f12304fb9e6ccad8646f682a"} Oct 31 15:19:58.503999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.503942 4865 scope.go:122] "RemoveContainer" containerID="690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b" Oct 31 15:19:58.505387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.505365 4865 generic.go:356] "Generic (PLEG): container finished" podID="81a79b66-503f-454d-b279-4be18a34ad2d" containerID="6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05" exitCode=0 Oct 31 15:19:58.505558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.505389 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" event={"ID":"81a79b66-503f-454d-b279-4be18a34ad2d","Type":"ContainerDied","Data":"6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05"} Oct 31 15:19:58.505558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.505415 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" Oct 31 15:19:58.505558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.505439 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg" event={"ID":"81a79b66-503f-454d-b279-4be18a34ad2d","Type":"ContainerDied","Data":"eb74894b30f61b3cf3bd910a515ed4620d3218a67acbd4ef91fa8f74edb6d8c5"} Oct 31 15:19:58.515253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.515238 4865 scope.go:122] "RemoveContainer" containerID="690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b" Oct 31 15:19:58.515558 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:58.515540 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b\": container with ID starting with 690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b not found: ID does not exist" containerID="690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b" Oct 31 15:19:58.515613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.515567 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b"} err="failed to get container status \"690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b\": rpc error: code = NotFound desc = could not find container \"690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b\": container with ID starting with 690f1032ac3ff2175a9b860700c34ee73ce89bc87499b58e76a85f9373f28d3b not found: ID does not exist" Oct 31 15:19:58.515613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.515583 4865 scope.go:122] "RemoveContainer" containerID="6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05" Oct 31 15:19:58.564277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.564243 4865 scope.go:122] "RemoveContainer" containerID="6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05" Oct 31 15:19:58.564740 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:19:58.564706 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05\": container with ID starting with 6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05 not found: ID does not exist" containerID="6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05" Oct 31 15:19:58.564883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.564759 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05"} err="failed to get container status \"6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05\": rpc error: code = NotFound desc = could not find container \"6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05\": container with ID starting with 6699b144f9cf5e07a838eec201a7314899a1b5aa055889fc212e1d7a08ba8c05 not found: ID does not exist" Oct 31 15:19:58.676276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.676186 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq"] Oct 31 15:19:58.679036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.679014 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-844949f94-n8xbq"] Oct 31 15:19:58.812176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.812144 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg"] Oct 31 15:19:58.814825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:19:58.814802 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-6cdbfb6df5-rwxlg"] Oct 31 15:20:00.201429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:00.201400 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:00.444185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:00.444159 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:20:00.444552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:00.444530 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:20:00.497203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:00.497176 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6230a2fd-0f3d-486c-b501-7228db16112e" path="/var/lib/kubelet/pods/6230a2fd-0f3d-486c-b501-7228db16112e/volumes" Oct 31 15:20:00.497608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:00.497594 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a79b66-503f-454d-b279-4be18a34ad2d" path="/var/lib/kubelet/pods/81a79b66-503f-454d-b279-4be18a34ad2d/volumes" Oct 31 15:20:02.063394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:02.063363 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:02.192021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:02.191985 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:03.602374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:03.602344 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:04.084573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:04.084542 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:04.105698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:04.105671 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8"] Oct 31 15:20:04.105890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:04.105870 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" containerID="cri-o://435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709" gracePeriod=30 Oct 31 15:20:04.413074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:04.413001 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:04.538461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:04.538415 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:04.883601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:04.883569 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:05.226670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:05.226633 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:05.943586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:05.943551 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:06.403332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:06.403305 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:06.409223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:06.409197 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8"] Oct 31 15:20:06.409426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:06.409404 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" containerID="cri-o://fba018447d234b4b0a91b9b353e5b678767af6d454ccc8f201bda29356ca68b7" gracePeriod=30 Oct 31 15:20:06.769655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:06.769625 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:08.861127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:08.861096 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:09.386983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.386922 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:20:09.388882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.388860 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6"] Oct 31 15:20:09.389445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389431 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" Oct 31 15:20:09.389524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389446 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" Oct 31 15:20:09.389524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389462 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" Oct 31 15:20:09.389524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389486 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" Oct 31 15:20:09.389524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389501 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6230a2fd-0f3d-486c-b501-7228db16112e" containerName="payload-processing" Oct 31 15:20:09.389524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389509 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6230a2fd-0f3d-486c-b501-7228db16112e" containerName="payload-processing" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389528 4865 cpu_manager.go:414] "containerMap: removing container" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389537 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389552 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389559 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389575 4865 cpu_manager.go:414] "containerMap: removing container" podUID="81a79b66-503f-454d-b279-4be18a34ad2d" containerName="payload-pre-processing" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389580 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a79b66-503f-454d-b279-4be18a34ad2d" containerName="payload-pre-processing" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389588 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389592 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389685 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="663838ff-9d0b-49d3-8cd4-62bf619754c8" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389695 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9113d241-1dd7-4d9f-a2af-c8043456e64b" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389705 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a79b66-503f-454d-b279-4be18a34ad2d" containerName="payload-pre-processing" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389711 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389721 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e312e76-afbd-49b9-82bf-c7a1d286bde9" containerName="istio-proxy" Oct 31 15:20:09.389728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.389727 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6230a2fd-0f3d-486c-b501-7228db16112e" containerName="payload-processing" Oct 31 15:20:09.393059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.393042 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.476529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.476499 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:09.478512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.478490 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d9cddd-openshift-default-dockercfg-7h672\"" Oct 31 15:20:09.496198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.496169 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6"] Oct 31 15:20:09.557771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.557737 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-data\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.557947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.557799 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istiod-ca-cert\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.557947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.557844 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-kube-api-access-rm5rj\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-kube-api-access-rm5rj\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.557947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.557932 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-token\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-token\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.558161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558024 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-socket\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.558161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558053 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-credential-socket\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.558161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558029 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-data" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:09.558339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558191 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istiod-ca-cert" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:09.558397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558342 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-credential-socket" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:09.558397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558360 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-certs\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.558540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558426 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-envoy\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.558540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558506 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-podinfo\") pod \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\" (UID: \"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4\") " Oct 31 15:20:09.558646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558537 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-socket" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:09.558730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558695 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-certs" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:09.558815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558759 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.558895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.558875 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.559108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.559081 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.559265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.559246 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.559423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.559281 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6gm\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-kube-api-access-bt6gm\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.559600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.559580 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14345f00-2d2a-4c48-b57f-93e63037a10d-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.559817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.559794 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.559926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.559836 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.559991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.559963 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.560081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.560062 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.560207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.560193 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.560313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.560293 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.560438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.560318 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.560438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.560332 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.561166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.561143 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-envoy" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:09.561271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.561172 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-kube-api-access-rm5rj" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "kube-api-access-rm5rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:09.561271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.561195 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-token" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:09.561394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.561372 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-podinfo" pod "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" (UID: "f85dc57d-13a8-4eec-b109-3ca2a2ad22f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:20:09.605970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.605943 4865 generic.go:356] "Generic (PLEG): container finished" podID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerID="435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709" exitCode=0 Oct 31 15:20:09.606069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.605995 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" Oct 31 15:20:09.606069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.606036 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" event={"ID":"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4","Type":"ContainerDied","Data":"435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709"} Oct 31 15:20:09.606152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.606094 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8" event={"ID":"f85dc57d-13a8-4eec-b109-3ca2a2ad22f4","Type":"ContainerDied","Data":"dc47f2d6e25095d14ecf11bd3289e35c449200bd1a042b4e311aaebdae8481fb"} Oct 31 15:20:09.606152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.606122 4865 scope.go:122] "RemoveContainer" containerID="435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709" Oct 31 15:20:09.621180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.620876 4865 scope.go:122] "RemoveContainer" containerID="406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666230 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666299 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666364 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666389 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6gm\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-kube-api-access-bt6gm\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666523 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14345f00-2d2a-4c48-b57f-93e63037a10d-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666565 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666595 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666726 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666763 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666870 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666887 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666904 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm5rj\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-kube-api-access-rm5rj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.667724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.666918 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:09.670491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.669548 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14345f00-2d2a-4c48-b57f-93e63037a10d-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.670491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.669804 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.670491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.670228 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.670491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.670402 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.677449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.671363 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.681378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.680567 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.683650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.682438 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.690920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.690893 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:09.715611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.715579 4865 scope.go:122] "RemoveContainer" containerID="435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709" Oct 31 15:20:09.717425 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:09.717352 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709\": container with ID starting with 435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709 not found: ID does not exist" containerID="435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709" Oct 31 15:20:09.717425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.717405 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709"} err="failed to get container status \"435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709\": rpc error: code = NotFound desc = could not find container \"435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709\": container with ID starting with 435237d685fa14c3adf6199b56abc0e3c8211a2b0164ea3e529759b0bb30c709 not found: ID does not exist" Oct 31 15:20:09.717794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.717432 4865 scope.go:122] "RemoveContainer" containerID="406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d" Oct 31 15:20:09.717946 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:09.717919 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d\": container with ID starting with 406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d not found: ID does not exist" containerID="406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d" Oct 31 15:20:09.718104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.718082 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d"} err="failed to get container status \"406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d\": rpc error: code = NotFound desc = could not find container \"406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d\": container with ID starting with 406a0bf2c36a9527e9cff0e709f9e9e69d491dc8cc17e8b45d5118ba48c2b40d not found: ID does not exist" Oct 31 15:20:09.733223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.733135 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.740060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.740008 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6gm\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-kube-api-access-bt6gm\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:09.750150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.750112 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8"] Oct 31 15:20:09.757150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:09.757116 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76cce-openshift-default-5cdcd6c445-wqjs8"] Oct 31 15:20:10.006008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.005981 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:10.205290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.205263 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:10.452203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.452178 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6"] Oct 31 15:20:10.452823 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:10.452799 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14345f00_2d2a_4c48_b57f_93e63037a10d.slice/crio-f90d99ae79fc7b87a8f60d59e0c9ca7ff274a2277bfd2c1cb1c9484a9fad8123 WatchSource:0}: Error finding container f90d99ae79fc7b87a8f60d59e0c9ca7ff274a2277bfd2c1cb1c9484a9fad8123: Status 404 returned error can't find the container with id f90d99ae79fc7b87a8f60d59e0c9ca7ff274a2277bfd2c1cb1c9484a9fad8123 Oct 31 15:20:10.455682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.455648 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:20:10.455780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.455712 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:20:10.455780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.455742 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:20:10.498076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.498047 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" path="/var/lib/kubelet/pods/f85dc57d-13a8-4eec-b109-3ca2a2ad22f4/volumes" Oct 31 15:20:10.617969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.617927 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" event={"ID":"14345f00-2d2a-4c48-b57f-93e63037a10d","Type":"ContainerStarted","Data":"4e740e2a31bd440e8ec13ae5040fd96b43a8c5bef6ff96d047b945beba15ae6b"} Oct 31 15:20:10.618157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.617995 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" event={"ID":"14345f00-2d2a-4c48-b57f-93e63037a10d","Type":"ContainerStarted","Data":"f90d99ae79fc7b87a8f60d59e0c9ca7ff274a2277bfd2c1cb1c9484a9fad8123"} Oct 31 15:20:10.738426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.738370 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podStartSLOduration=1.738355269 podStartE2EDuration="1.738355269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:10.736915458 +0000 UTC m=+2056.356466883" watchObservedRunningTime="2026-10-31 15:20:10.738355269 +0000 UTC m=+2056.357906680" Oct 31 15:20:10.796320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.796283 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:10.964347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:10.964320 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:11.006424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.006341 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:11.011823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.011799 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:11.631458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.631428 4865 generic.go:356] "Generic (PLEG): container finished" podID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerID="fba018447d234b4b0a91b9b353e5b678767af6d454ccc8f201bda29356ca68b7" exitCode=0 Oct 31 15:20:11.631672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.631525 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" event={"ID":"f12211dd-e275-441c-8e43-088e4c2bc3af","Type":"ContainerDied","Data":"fba018447d234b4b0a91b9b353e5b678767af6d454ccc8f201bda29356ca68b7"} Oct 31 15:20:11.631672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.631599 4865 scope.go:122] "RemoveContainer" containerID="f47a32101578a89e75362edbedcc883cca0aeb339c3f4a64e69d564b36859ed3" Oct 31 15:20:11.631962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.631945 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:11.633389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.633371 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:20:11.687169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.687121 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:20:11.794765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794735 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-socket\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.794988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794793 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-certs\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.794988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794824 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-kube-api-access-5r7md\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-kube-api-access-5r7md\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.794988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794857 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-credential-socket\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.794988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794876 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-data\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.794988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794919 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-token\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-token\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.794988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794966 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12211dd-e275-441c-8e43-088e4c2bc3af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12211dd-e275-441c-8e43-088e4c2bc3af-istiod-ca-cert\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.795325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.794999 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-podinfo\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.795325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.795046 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-socket" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:11.795325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.795182 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-certs" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:11.795325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.795199 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-credential-socket" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:11.795325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.795213 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-data" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:11.795325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.795323 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-envoy\") pod \"f12211dd-e275-441c-8e43-088e4c2bc3af\" (UID: \"f12211dd-e275-441c-8e43-088e4c2bc3af\") " Oct 31 15:20:11.795697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.795376 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12211dd-e275-441c-8e43-088e4c2bc3af-istiod-ca-cert" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:11.796240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.796215 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.796240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.796237 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.796400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.796253 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.796400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.796265 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.796400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.796280 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12211dd-e275-441c-8e43-088e4c2bc3af-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.797681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.797659 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-podinfo" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:20:11.797929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.797908 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-kube-api-access-5r7md" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "kube-api-access-5r7md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:11.797995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.797932 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-envoy" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:11.797995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.797943 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-token" pod "f12211dd-e275-441c-8e43-088e4c2bc3af" (UID: "f12211dd-e275-441c-8e43-088e4c2bc3af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:11.897448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.897413 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r7md\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-kube-api-access-5r7md\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.897626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.897452 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.897626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.897484 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.897626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.897502 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12211dd-e275-441c-8e43-088e4c2bc3af-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:11.950202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:11.950123 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh"] Oct 31 15:20:12.117984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.117953 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:12.303591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.303548 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:12.437500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.437439 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:12.701233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.701143 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" Oct 31 15:20:12.704530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.701813 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8" event={"ID":"f12211dd-e275-441c-8e43-088e4c2bc3af","Type":"ContainerDied","Data":"df9bb29e6ab239eb1bb9b34f55900768ce5ec334b35d19077b49aae58099d745"} Oct 31 15:20:12.704530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.701873 4865 scope.go:122] "RemoveContainer" containerID="fba018447d234b4b0a91b9b353e5b678767af6d454ccc8f201bda29356ca68b7" Oct 31 15:20:12.841566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.841531 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx"] Oct 31 15:20:12.842175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.842155 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" Oct 31 15:20:12.842296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.842178 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" Oct 31 15:20:12.842296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.842228 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" Oct 31 15:20:12.842296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.842236 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" Oct 31 15:20:12.842460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.842387 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" Oct 31 15:20:12.842460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.842411 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85dc57d-13a8-4eec-b109-3ca2a2ad22f4" containerName="istio-proxy" Oct 31 15:20:12.847119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.847099 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:12.875806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.875783 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8"] Oct 31 15:20:12.881620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.881598 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-94a889-openshift-default-676f4b7659-sc6g8"] Oct 31 15:20:12.904868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.904841 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx"] Oct 31 15:20:12.920025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.920001 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:12.938521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:12.938462 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-15cc44-openshift-default-dockercfg-7lwjw\"" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014540 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-socket\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014588 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2sz\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-kube-api-access-jj2sz\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014630 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-data\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014676 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-credential-socket\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014779 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-token\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014836 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-podinfo\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014882 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539e7c13-c6e7-49ad-a03d-617a87a5157b-istiod-ca-cert\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014944 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-envoy\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.015493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.014967 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-certs\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.043412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.043362 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:13.116222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-podinfo\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116245 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539e7c13-c6e7-49ad-a03d-617a87a5157b-istiod-ca-cert\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116314 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-envoy\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116340 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-certs\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116443 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-socket\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116507 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2sz\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-kube-api-access-jj2sz\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116549 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-data\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116595 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-credential-socket\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116707 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-token\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116848 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-certs\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.116941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.116910 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-socket\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.117106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.117080 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539e7c13-c6e7-49ad-a03d-617a87a5157b-istiod-ca-cert\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.117106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.117096 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-data\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.117353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.117330 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-credential-socket\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.119212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.119183 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-podinfo\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.119576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.119452 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-envoy\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.174551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.174527 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2sz\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-kube-api-access-jj2sz\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.175017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.175000 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-token\") pod \"e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.251999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.251968 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:13.461058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.461023 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:13.484126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.484092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:13.729518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.720665 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:13.909864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.909833 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx"] Oct 31 15:20:13.940697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.940642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:13.940806 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:13.940718 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539e7c13_c6e7_49ad_a03d_617a87a5157b.slice/crio-0e369112ab34a6fbea610938330b66cdc8d31550a0f1ba6b713e21bcebee423a WatchSource:0}: Error finding container 0e369112ab34a6fbea610938330b66cdc8d31550a0f1ba6b713e21bcebee423a: Status 404 returned error can't find the container with id 0e369112ab34a6fbea610938330b66cdc8d31550a0f1ba6b713e21bcebee423a Oct 31 15:20:13.943094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.943068 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:20:13.943179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.943127 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:20:13.943179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:13.943156 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:20:14.499438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:14.499409 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" path="/var/lib/kubelet/pods/f12211dd-e275-441c-8e43-088e4c2bc3af/volumes" Oct 31 15:20:14.730985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:14.730955 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" event={"ID":"539e7c13-c6e7-49ad-a03d-617a87a5157b","Type":"ContainerStarted","Data":"e28b2fd24e44a296f5bf2ae4f7863d0f13e872ec7733327d4686a6625af67cd0"} Oct 31 15:20:14.731148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:14.731014 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" event={"ID":"539e7c13-c6e7-49ad-a03d-617a87a5157b","Type":"ContainerStarted","Data":"0e369112ab34a6fbea610938330b66cdc8d31550a0f1ba6b713e21bcebee423a"} Oct 31 15:20:14.743515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:14.743488 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:14.904504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:14.904356 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podStartSLOduration=2.904337056 podStartE2EDuration="2.904337056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:14.900601234 +0000 UTC m=+2060.520152646" watchObservedRunningTime="2026-10-31 15:20:14.904337056 +0000 UTC m=+2060.523888467" Oct 31 15:20:15.461864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:15.461828 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:15.467057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:15.467035 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:15.782954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:15.782868 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:15.784077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:15.784050 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:20:16.529397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:16.529369 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:16.663018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:16.662985 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:16.791510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:16.791405 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:16.950928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:16.950889 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" podUID="920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" containerName="payload-processing" containerID="cri-o://3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4" gracePeriod=25 Oct 31 15:20:17.262260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.262227 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:20:17.298111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.298085 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-kube-api-access-z47j4\" (UniqueName: \"kubernetes.io/projected/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-kube-api-access-z47j4\") pod \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " Oct 31 15:20:17.298268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.298128 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-plugins-config-volume\") pod \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\" (UID: \"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15\") " Oct 31 15:20:17.298599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.298575 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-plugins-config-volume" pod "920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" (UID: "920b0860-9a3f-4fa0-a26d-5aca8ef4bd15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:17.300410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.300392 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-kube-api-access-z47j4" pod "920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" (UID: "920b0860-9a3f-4fa0-a26d-5aca8ef4bd15"). InnerVolumeSpecName "kube-api-access-z47j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:17.379908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.379817 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:20:17.400080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.400044 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z47j4\" (UniqueName: \"kubernetes.io/projected/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-kube-api-access-z47j4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:17.400080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.400076 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:17.596047 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:17.596011 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:20:17.803368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.803333 4865 generic.go:356] "Generic (PLEG): container finished" podID="920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" containerID="3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4" exitCode=0 Oct 31 15:20:17.803368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.803378 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" Oct 31 15:20:17.803922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.803424 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" event={"ID":"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15","Type":"ContainerDied","Data":"3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4"} Oct 31 15:20:17.803922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.803492 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh" event={"ID":"920b0860-9a3f-4fa0-a26d-5aca8ef4bd15","Type":"ContainerDied","Data":"f2d33f5eff650f1b470a56daf259563c203f5a76ecfdc8a2774a00db78373706"} Oct 31 15:20:17.803922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.803519 4865 scope.go:122] "RemoveContainer" containerID="3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4" Oct 31 15:20:17.814105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.814084 4865 scope.go:122] "RemoveContainer" containerID="3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4" Oct 31 15:20:17.814407 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:17.814388 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4\": container with ID starting with 3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4 not found: ID does not exist" containerID="3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4" Oct 31 15:20:17.814480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.814414 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4"} err="failed to get container status \"3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4\": rpc error: code = NotFound desc = could not find container \"3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4\": container with ID starting with 3d1a8c8568e0d08ec0a945d4cafe9a91c43acfda7cd143c757cdadaf0f48eab4 not found: ID does not exist" Oct 31 15:20:17.970574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:17.970545 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh"] Oct 31 15:20:18.021644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:18.021616 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9cc557-64dc88996f-tbvfh"] Oct 31 15:20:18.353026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:18.352990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:18.496739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:18.496711 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" path="/var/lib/kubelet/pods/920b0860-9a3f-4fa0-a26d-5aca8ef4bd15/volumes" Oct 31 15:20:18.837534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:18.837503 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:18.849611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:18.849586 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm"] Oct 31 15:20:18.849808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:18.849789 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" containerID="cri-o://ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f" gracePeriod=30 Oct 31 15:20:18.969075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:18.969028 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:19.108600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:19.108461 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:19.812839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:19.812808 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:20.431492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.431413 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:20.585534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.580083 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:20.737493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.725426 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:20.833165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.833136 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b"] Oct 31 15:20:20.834324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.834296 4865 cpu_manager.go:414] "containerMap: removing container" podUID="920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" containerName="payload-processing" Oct 31 15:20:20.834548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.834535 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" containerName="payload-processing" Oct 31 15:20:20.834817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.834803 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12211dd-e275-441c-8e43-088e4c2bc3af" containerName="istio-proxy" Oct 31 15:20:20.834922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.834911 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="920b0860-9a3f-4fa0-a26d-5aca8ef4bd15" containerName="payload-processing" Oct 31 15:20:20.840382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.840360 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:20.895081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.895050 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b"] Oct 31 15:20:20.990549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.967864 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d9cddd-dockercfg-wkzvh\"" Oct 31 15:20:20.990549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:20.967949 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d9cddd\"" Oct 31 15:20:21.005504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.002454 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f8385-144a-44f8-98fb-e0bec936f0ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:21.005504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.002679 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5dr\" (UniqueName: \"kubernetes.io/projected/cc6f8385-144a-44f8-98fb-e0bec936f0ff-kube-api-access-lj5dr\") pod \"payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:21.012203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.012170 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8"] Oct 31 15:20:21.019151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.019114 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.055586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.055557 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8"] Oct 31 15:20:21.104055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.104020 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f8385-144a-44f8-98fb-e0bec936f0ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:21.104266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.104125 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70601-d132-4771-a7a5-f9e9080ac2bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.104266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.104175 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5dr\" (UniqueName: \"kubernetes.io/projected/cc6f8385-144a-44f8-98fb-e0bec936f0ff-kube-api-access-lj5dr\") pod \"payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:21.104403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.104303 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6zb\" (UniqueName: \"kubernetes.io/projected/40d70601-d132-4771-a7a5-f9e9080ac2bb-kube-api-access-rl6zb\") pod \"payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.104881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.104856 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f8385-144a-44f8-98fb-e0bec936f0ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:21.162319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.162289 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5dr\" (UniqueName: \"kubernetes.io/projected/cc6f8385-144a-44f8-98fb-e0bec936f0ff-kube-api-access-lj5dr\") pod \"payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:21.205686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.205653 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70601-d132-4771-a7a5-f9e9080ac2bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.205907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.205754 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6zb\" (UniqueName: \"kubernetes.io/projected/40d70601-d132-4771-a7a5-f9e9080ac2bb-kube-api-access-rl6zb\") pod \"payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.206403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.206378 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70601-d132-4771-a7a5-f9e9080ac2bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.264287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.264215 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6zb\" (UniqueName: \"kubernetes.io/projected/40d70601-d132-4771-a7a5-f9e9080ac2bb-kube-api-access-rl6zb\") pod \"payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.341545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.341516 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.454545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.454510 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:21.714917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.714886 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8"] Oct 31 15:20:21.715342 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:21.715316 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d70601_d132_4771_a7a5_f9e9080ac2bb.slice/crio-ed1dee29eeef826670225f46142b69a897bbfcaf42e9eb97a2db0487debadc07 WatchSource:0}: Error finding container ed1dee29eeef826670225f46142b69a897bbfcaf42e9eb97a2db0487debadc07: Status 404 returned error can't find the container with id ed1dee29eeef826670225f46142b69a897bbfcaf42e9eb97a2db0487debadc07 Oct 31 15:20:21.838672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.838644 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b"] Oct 31 15:20:21.838870 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:21.838847 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6f8385_144a_44f8_98fb_e0bec936f0ff.slice/crio-b95fa5145b99046fa6dd67c7fe03564564cc0e058c7292f7fbd2b8bdec594351 WatchSource:0}: Error finding container b95fa5145b99046fa6dd67c7fe03564564cc0e058c7292f7fbd2b8bdec594351: Status 404 returned error can't find the container with id b95fa5145b99046fa6dd67c7fe03564564cc0e058c7292f7fbd2b8bdec594351 Oct 31 15:20:21.845613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.845585 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" event={"ID":"40d70601-d132-4771-a7a5-f9e9080ac2bb","Type":"ContainerStarted","Data":"51ac34c4b8c3f45df60a78cacd1eff49b33fa6960ae5feb4d11795c53ee760f1"} Oct 31 15:20:21.845745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.845637 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" event={"ID":"40d70601-d132-4771-a7a5-f9e9080ac2bb","Type":"ContainerStarted","Data":"ed1dee29eeef826670225f46142b69a897bbfcaf42e9eb97a2db0487debadc07"} Oct 31 15:20:21.845745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.845681 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:21.961973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:21.961921 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" podStartSLOduration=1.961908577 podStartE2EDuration="1.961908577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:21.960387137 +0000 UTC m=+2067.579938549" watchObservedRunningTime="2026-10-31 15:20:21.961908577 +0000 UTC m=+2067.581459988" Oct 31 15:20:22.028780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:22.028718 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:22.854041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:22.854007 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" event={"ID":"cc6f8385-144a-44f8-98fb-e0bec936f0ff","Type":"ContainerStarted","Data":"d2b50fc65cf831f1cf84a99982c7c196b5b31e688949e9efc9b75d1276939689"} Oct 31 15:20:22.854443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:22.854069 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" event={"ID":"cc6f8385-144a-44f8-98fb-e0bec936f0ff","Type":"ContainerStarted","Data":"b95fa5145b99046fa6dd67c7fe03564564cc0e058c7292f7fbd2b8bdec594351"} Oct 31 15:20:22.854443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:22.854099 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:22.928830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:22.928796 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:22.970072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:22.970022 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" podStartSLOduration=2.970009204 podStartE2EDuration="2.970009204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:22.968544141 +0000 UTC m=+2068.588095556" watchObservedRunningTime="2026-10-31 15:20:22.970009204 +0000 UTC m=+2068.589560614" Oct 31 15:20:23.121398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.121297 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:23.350843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.350808 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw"] Oct 31 15:20:23.355314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.355295 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.421368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.421270 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-15cc44\"" Oct 31 15:20:23.436628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.436567 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw"] Oct 31 15:20:23.461420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.461374 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952fn\" (UniqueName: \"kubernetes.io/projected/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-kube-api-access-952fn\") pod \"payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.461800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.461777 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.472781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.472749 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-15cc44-dockercfg-qqs4n\"" Oct 31 15:20:23.530269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.530212 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj"] Oct 31 15:20:23.535866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.535845 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:23.564854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.564826 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj"] Oct 31 15:20:23.565801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.565770 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952fn\" (UniqueName: \"kubernetes.io/projected/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-kube-api-access-952fn\") pod \"payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.565946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.565865 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.566542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.566523 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.623874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.623843 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952fn\" (UniqueName: \"kubernetes.io/projected/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-kube-api-access-952fn\") pod \"payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.666973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.666938 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfssb\" (UniqueName: \"kubernetes.io/projected/93677797-db5f-4973-9171-f01fbe7d20fd-kube-api-access-bfssb\") pod \"payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:23.666973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.666976 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93677797-db5f-4973-9171-f01fbe7d20fd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:23.669660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.669639 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:23.768364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.768327 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfssb\" (UniqueName: \"kubernetes.io/projected/93677797-db5f-4973-9171-f01fbe7d20fd-kube-api-access-bfssb\") pod \"payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:23.768571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.768377 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93677797-db5f-4973-9171-f01fbe7d20fd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:23.769151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.769127 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93677797-db5f-4973-9171-f01fbe7d20fd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:23.827953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.827918 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfssb\" (UniqueName: \"kubernetes.io/projected/93677797-db5f-4973-9171-f01fbe7d20fd-kube-api-access-bfssb\") pod \"payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:23.854943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:23.854917 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:24.045440 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:24.045396 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109b3301_6d14_4a0b_8cd7_e25377ef3dbd.slice/crio-bb81d9e40dc95c8f468a4b05ef8626d2bbb7d752cde523c09957a960762dbc98 WatchSource:0}: Error finding container bb81d9e40dc95c8f468a4b05ef8626d2bbb7d752cde523c09957a960762dbc98: Status 404 returned error can't find the container with id bb81d9e40dc95c8f468a4b05ef8626d2bbb7d752cde523c09957a960762dbc98 Oct 31 15:20:24.047533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.047510 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw"] Oct 31 15:20:24.125009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.124988 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:20:24.235163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.235129 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj"] Oct 31 15:20:24.235530 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:24.235507 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93677797_db5f_4973_9171_f01fbe7d20fd.slice/crio-6357f88556a40dc05440ddb2ed9b262fb64cf0b8f35d518230540c0c46fd8ae4 WatchSource:0}: Error finding container 6357f88556a40dc05440ddb2ed9b262fb64cf0b8f35d518230540c0c46fd8ae4: Status 404 returned error can't find the container with id 6357f88556a40dc05440ddb2ed9b262fb64cf0b8f35d518230540c0c46fd8ae4 Oct 31 15:20:24.275233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275204 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-podinfo\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275249 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-kube-api-access-xpglh\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-kube-api-access-xpglh\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275307 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-credential-socket\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275366 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-envoy\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275438 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-token\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-token\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275552 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d091acc-c288-43e4-99dd-a505a4f62fb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d091acc-c288-43e4-99dd-a505a4f62fb0-istiod-ca-cert\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275686 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-data\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275729 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-socket\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275772 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-certs\") pod \"5d091acc-c288-43e4-99dd-a505a4f62fb0\" (UID: \"5d091acc-c288-43e4-99dd-a505a4f62fb0\") " Oct 31 15:20:24.275918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.275889 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-credential-socket" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:24.276140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.276110 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-certs" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:24.276692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.276504 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d091acc-c288-43e4-99dd-a505a4f62fb0-istiod-ca-cert" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:24.276795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.276717 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-data" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:24.277060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.277023 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-socket" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:24.277413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.277361 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d091acc-c288-43e4-99dd-a505a4f62fb0-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.277413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.277390 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.277413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.277405 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.277850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.277419 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.277850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.277434 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.278176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.278148 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-podinfo" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:20:24.280396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.280375 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-kube-api-access-xpglh" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "kube-api-access-xpglh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:24.280559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.280445 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-token" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:24.280624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.280557 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-envoy" pod "5d091acc-c288-43e4-99dd-a505a4f62fb0" (UID: "5d091acc-c288-43e4-99dd-a505a4f62fb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:24.362147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.362121 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:24.378684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.378661 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.378684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.378680 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpglh\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-kube-api-access-xpglh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.378818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.378692 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.378818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.378701 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d091acc-c288-43e4-99dd-a505a4f62fb0-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:24.534982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.534952 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:24.668359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.668325 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:24.869539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.869432 4865 generic.go:356] "Generic (PLEG): container finished" podID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerID="ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f" exitCode=0 Oct 31 15:20:24.869539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.869504 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" Oct 31 15:20:24.869539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.869509 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" event={"ID":"5d091acc-c288-43e4-99dd-a505a4f62fb0","Type":"ContainerDied","Data":"ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f"} Oct 31 15:20:24.870105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.869593 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm" event={"ID":"5d091acc-c288-43e4-99dd-a505a4f62fb0","Type":"ContainerDied","Data":"02c1b4d76db8b9f6c76a981973458f0f4a483dc79db8b30fe3aea95df495c71a"} Oct 31 15:20:24.870105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.869618 4865 scope.go:122] "RemoveContainer" containerID="ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f" Oct 31 15:20:24.871416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.871392 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" event={"ID":"93677797-db5f-4973-9171-f01fbe7d20fd","Type":"ContainerStarted","Data":"0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694"} Oct 31 15:20:24.871565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.871441 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" event={"ID":"93677797-db5f-4973-9171-f01fbe7d20fd","Type":"ContainerStarted","Data":"6357f88556a40dc05440ddb2ed9b262fb64cf0b8f35d518230540c0c46fd8ae4"} Oct 31 15:20:24.871693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.871679 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:24.873253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.873226 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" event={"ID":"109b3301-6d14-4a0b-8cd7-e25377ef3dbd","Type":"ContainerStarted","Data":"08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a"} Oct 31 15:20:24.873353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.873274 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" event={"ID":"109b3301-6d14-4a0b-8cd7-e25377ef3dbd","Type":"ContainerStarted","Data":"bb81d9e40dc95c8f468a4b05ef8626d2bbb7d752cde523c09957a960762dbc98"} Oct 31 15:20:24.873431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.873398 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:24.880802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.880789 4865 scope.go:122] "RemoveContainer" containerID="54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f" Oct 31 15:20:24.889919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.889904 4865 scope.go:122] "RemoveContainer" containerID="ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f" Oct 31 15:20:24.890192 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:24.890176 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f\": container with ID starting with ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f not found: ID does not exist" containerID="ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f" Oct 31 15:20:24.890238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.890207 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f"} err="failed to get container status \"ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f\": rpc error: code = NotFound desc = could not find container \"ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f\": container with ID starting with ec8c770b1afbf3b51d3620f64534a9e04bc0bc2f66d0a4ce1521b8eb2adaa62f not found: ID does not exist" Oct 31 15:20:24.890238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.890223 4865 scope.go:122] "RemoveContainer" containerID="54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f" Oct 31 15:20:24.890457 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:24.890439 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f\": container with ID starting with 54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f not found: ID does not exist" containerID="54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f" Oct 31 15:20:24.890636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:24.890461 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f"} err="failed to get container status \"54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f\": rpc error: code = NotFound desc = could not find container \"54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f\": container with ID starting with 54e85f649ee99eb0786af54b0f952735a0015c317649532f49951086cfa5d93f not found: ID does not exist" Oct 31 15:20:25.040439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:25.040385 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" podStartSLOduration=2.040369804 podStartE2EDuration="2.040369804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:25.038328383 +0000 UTC m=+2070.657879794" watchObservedRunningTime="2026-10-31 15:20:25.040369804 +0000 UTC m=+2070.659921228" Oct 31 15:20:25.173545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:25.172896 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" podStartSLOduration=2.172875975 podStartE2EDuration="2.172875975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:25.171144419 +0000 UTC m=+2070.790695833" watchObservedRunningTime="2026-10-31 15:20:25.172875975 +0000 UTC m=+2070.792427388" Oct 31 15:20:25.296956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:25.296912 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm"] Oct 31 15:20:25.300875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:25.300765 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9cc557-openshift-default-5bbfc44d9f-2z9jm"] Oct 31 15:20:26.498319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:26.498285 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" path="/var/lib/kubelet/pods/5d091acc-c288-43e4-99dd-a505a4f62fb0/volumes" Oct 31 15:20:27.455649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:27.455613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:29.985937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:29.985910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:30.174914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.174322 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:30.411676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.411593 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467"] Oct 31 15:20:30.412261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.412242 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" Oct 31 15:20:30.412380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.412263 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" Oct 31 15:20:30.412380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.412312 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" Oct 31 15:20:30.412380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.412320 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" Oct 31 15:20:30.412597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.412531 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" Oct 31 15:20:30.412597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.412553 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d091acc-c288-43e4-99dd-a505a4f62fb0" containerName="istio-proxy" Oct 31 15:20:30.417616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.417597 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.464490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.464450 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.464614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.464560 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s74j\" (UniqueName: \"kubernetes.io/projected/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-kube-api-access-5s74j\") pod \"payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.471663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.471634 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467"] Oct 31 15:20:30.563929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.563503 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-da9920-dockercfg-b9q94\"" Oct 31 15:20:30.563929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.563627 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-da9920\"" Oct 31 15:20:30.591735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.591696 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.592043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.591797 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s74j\" (UniqueName: \"kubernetes.io/projected/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-kube-api-access-5s74j\") pod \"payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.599524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.599491 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.616862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.616352 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m"] Oct 31 15:20:30.621599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.621561 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:30.634840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.634811 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m"] Oct 31 15:20:30.653767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.653737 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s74j\" (UniqueName: \"kubernetes.io/projected/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-kube-api-access-5s74j\") pod \"payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.693640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.693566 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:30.693640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.693636 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29rb\" (UniqueName: \"kubernetes.io/projected/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-kube-api-access-m29rb\") pod \"payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:30.731395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.731363 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:30.802443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.802412 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:30.802633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.802511 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29rb\" (UniqueName: \"kubernetes.io/projected/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-kube-api-access-m29rb\") pod \"payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:30.803350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.803322 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:30.860557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.860521 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29rb\" (UniqueName: \"kubernetes.io/projected/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-kube-api-access-m29rb\") pod \"payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:30.937830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:30.937802 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:31.115424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.115393 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467"] Oct 31 15:20:31.115910 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:31.115876 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecf4477_0cb1_4fa5_8eaf_a9289b0d605a.slice/crio-01207a82d637f78c0c031a57e61f5874b0f29c1491e20c746773f019acf9e853 WatchSource:0}: Error finding container 01207a82d637f78c0c031a57e61f5874b0f29c1491e20c746773f019acf9e853: Status 404 returned error can't find the container with id 01207a82d637f78c0c031a57e61f5874b0f29c1491e20c746773f019acf9e853 Oct 31 15:20:31.314835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.314808 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m"] Oct 31 15:20:31.315421 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:20:31.315322 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e2069a_ae4b_44c1_91df_b6e8ca86238b.slice/crio-ce7019e63d8be4d4c7b16d5c3bc8b83ee353698ebf390d4842911f608f9c465d WatchSource:0}: Error finding container ce7019e63d8be4d4c7b16d5c3bc8b83ee353698ebf390d4842911f608f9c465d: Status 404 returned error can't find the container with id ce7019e63d8be4d4c7b16d5c3bc8b83ee353698ebf390d4842911f608f9c465d Oct 31 15:20:31.680282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.680197 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:31.922370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.922325 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" event={"ID":"e6e2069a-ae4b-44c1-91df-b6e8ca86238b","Type":"ContainerStarted","Data":"92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a"} Oct 31 15:20:31.922597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.922392 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" event={"ID":"e6e2069a-ae4b-44c1-91df-b6e8ca86238b","Type":"ContainerStarted","Data":"ce7019e63d8be4d4c7b16d5c3bc8b83ee353698ebf390d4842911f608f9c465d"} Oct 31 15:20:31.922597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.922560 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:31.924016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.923992 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" event={"ID":"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a","Type":"ContainerStarted","Data":"3c853e3569fa2901b5fa4ed25bf3361a140af9a85448f12031a0cea9db611b0e"} Oct 31 15:20:31.924128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.924033 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" event={"ID":"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a","Type":"ContainerStarted","Data":"01207a82d637f78c0c031a57e61f5874b0f29c1491e20c746773f019acf9e853"} Oct 31 15:20:31.924128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:31.924080 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:32.060213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.060155 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" podStartSLOduration=2.060139242 podStartE2EDuration="2.060139242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:32.039875761 +0000 UTC m=+2077.659427172" watchObservedRunningTime="2026-10-31 15:20:32.060139242 +0000 UTC m=+2077.679690653" Oct 31 15:20:32.063331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.063297 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh"] Oct 31 15:20:32.063578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.063554 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" podUID="7a20fe9f-e754-480f-b136-70ce4beee1c3" containerName="payload-processing" containerID="cri-o://34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131" gracePeriod=30 Oct 31 15:20:32.113593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.113568 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv"] Oct 31 15:20:32.113764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.113746 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" podUID="171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" containerName="payload-pre-processing" containerID="cri-o://94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742" gracePeriod=30 Oct 31 15:20:32.203030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.203001 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:32.291565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.291509 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" podStartSLOduration=2.291493786 podStartE2EDuration="2.291493786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:32.172607468 +0000 UTC m=+2077.792158890" watchObservedRunningTime="2026-10-31 15:20:32.291493786 +0000 UTC m=+2077.911045211" Oct 31 15:20:32.321517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.321394 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:32.356583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.356547 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:20:32.380891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.380869 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:20:32.443869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.443838 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a20fe9f-e754-480f-b136-70ce4beee1c3-kube-api-access-pllz7\" (UniqueName: \"kubernetes.io/projected/7a20fe9f-e754-480f-b136-70ce4beee1c3-kube-api-access-pllz7\") pod \"7a20fe9f-e754-480f-b136-70ce4beee1c3\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " Oct 31 15:20:32.444096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.444078 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-kube-api-access-65mln\" (UniqueName: \"kubernetes.io/projected/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-kube-api-access-65mln\") pod \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " Oct 31 15:20:32.444139 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.444112 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a20fe9f-e754-480f-b136-70ce4beee1c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a20fe9f-e754-480f-b136-70ce4beee1c3-plugins-config-volume\") pod \"7a20fe9f-e754-480f-b136-70ce4beee1c3\" (UID: \"7a20fe9f-e754-480f-b136-70ce4beee1c3\") " Oct 31 15:20:32.444172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.444142 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-plugins-config-volume\") pod \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\" (UID: \"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9\") " Oct 31 15:20:32.444566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.444538 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-plugins-config-volume" pod "171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" (UID: "171ca979-9d95-4ead-b4a6-f6a5ff8e14b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:32.444677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.444538 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a20fe9f-e754-480f-b136-70ce4beee1c3-plugins-config-volume" pod "7a20fe9f-e754-480f-b136-70ce4beee1c3" (UID: "7a20fe9f-e754-480f-b136-70ce4beee1c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:32.444716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.444693 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a20fe9f-e754-480f-b136-70ce4beee1c3-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.444716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.444708 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.446107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.446087 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a20fe9f-e754-480f-b136-70ce4beee1c3-kube-api-access-pllz7" pod "7a20fe9f-e754-480f-b136-70ce4beee1c3" (UID: "7a20fe9f-e754-480f-b136-70ce4beee1c3"). InnerVolumeSpecName "kube-api-access-pllz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:32.446183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.446151 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-kube-api-access-65mln" pod "171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" (UID: "171ca979-9d95-4ead-b4a6-f6a5ff8e14b9"). InnerVolumeSpecName "kube-api-access-65mln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:32.546346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.546318 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65mln\" (UniqueName: \"kubernetes.io/projected/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9-kube-api-access-65mln\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.547096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.547071 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pllz7\" (UniqueName: \"kubernetes.io/projected/7a20fe9f-e754-480f-b136-70ce4beee1c3-kube-api-access-pllz7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.857183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.857145 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:20:32.931954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.931872 4865 generic.go:356] "Generic (PLEG): container finished" podID="171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" containerID="94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742" exitCode=0 Oct 31 15:20:32.931954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.931917 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" Oct 31 15:20:32.932175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.931956 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" event={"ID":"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9","Type":"ContainerDied","Data":"94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742"} Oct 31 15:20:32.932175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.932004 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv" event={"ID":"171ca979-9d95-4ead-b4a6-f6a5ff8e14b9","Type":"ContainerDied","Data":"17abfdcc7322909d4349c7363403809dba4ecadff135d6c61c4f7224fe9e1296"} Oct 31 15:20:32.932175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.932038 4865 scope.go:122] "RemoveContainer" containerID="94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742" Oct 31 15:20:32.933518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.933494 4865 generic.go:356] "Generic (PLEG): container finished" podID="7a20fe9f-e754-480f-b136-70ce4beee1c3" containerID="34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131" exitCode=0 Oct 31 15:20:32.933658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.933531 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" Oct 31 15:20:32.933658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.933578 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" event={"ID":"7a20fe9f-e754-480f-b136-70ce4beee1c3","Type":"ContainerDied","Data":"34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131"} Oct 31 15:20:32.933658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.933628 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh" event={"ID":"7a20fe9f-e754-480f-b136-70ce4beee1c3","Type":"ContainerDied","Data":"6a2ceecb68a6bb0b78f54e957ff283630542aab7f600256f31acfbb10416493d"} Oct 31 15:20:32.942505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.942463 4865 scope.go:122] "RemoveContainer" containerID="94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742" Oct 31 15:20:32.942773 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:32.942756 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742\": container with ID starting with 94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742 not found: ID does not exist" containerID="94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742" Oct 31 15:20:32.942821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.942781 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742"} err="failed to get container status \"94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742\": rpc error: code = NotFound desc = could not find container \"94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742\": container with ID starting with 94ff5f8d30e8152eee8a93f86533db09b75959c036b0fcc8dc88ce62612b7742 not found: ID does not exist" Oct 31 15:20:32.942821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.942799 4865 scope.go:122] "RemoveContainer" containerID="34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131" Oct 31 15:20:32.951524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.951505 4865 scope.go:122] "RemoveContainer" containerID="34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131" Oct 31 15:20:32.951786 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:32.951768 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131\": container with ID starting with 34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131 not found: ID does not exist" containerID="34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131" Oct 31 15:20:32.951855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:32.951796 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131"} err="failed to get container status \"34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131\": rpc error: code = NotFound desc = could not find container \"34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131\": container with ID starting with 34a0607947319512bacac23bd799f32688580d6f636c6faa99a7e101b782e131 not found: ID does not exist" Oct 31 15:20:33.091621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:33.091591 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh"] Oct 31 15:20:33.094425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:33.094402 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-032b05-7dcbcc9d4b-6f2nh"] Oct 31 15:20:33.201283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:33.201201 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv"] Oct 31 15:20:33.204178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:33.204158 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-032b05-789bf59787-lpxgv"] Oct 31 15:20:33.486561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:33.486530 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:33.863822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:33.863747 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:20:34.498061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:34.498033 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" path="/var/lib/kubelet/pods/171ca979-9d95-4ead-b4a6-f6a5ff8e14b9/volumes" Oct 31 15:20:34.498461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:34.498434 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a20fe9f-e754-480f-b136-70ce4beee1c3" path="/var/lib/kubelet/pods/7a20fe9f-e754-480f-b136-70ce4beee1c3/volumes" Oct 31 15:20:35.748721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:35.748685 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:35.884395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:35.884369 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:20:35.884602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:35.884431 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:20:36.298315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:36.298283 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:36.425726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:36.425692 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:37.859264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:37.859227 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:38.412178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:38.412145 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:38.662532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:38.662431 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:39.317672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:39.317632 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:42.937020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:42.936992 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:20:42.937591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:42.937056 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:20:44.917991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:44.917961 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:45.385925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:45.385887 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:47.491176 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:20:47.491140 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:20:50.527050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:50.527007 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:54.906033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:54.906004 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:20:54.907253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:54.907231 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:20:54.923173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:54.923146 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:20:54.924270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:54.924248 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:20:55.559132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:55.559089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:56.607763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:56.607726 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:56.920690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:56.920659 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:20:58.005197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:20:58.005162 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:00.329367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:00.329334 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:00.604623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:00.604549 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:04.067737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:04.067695 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:05.307136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:05.307059 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:05.948074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:05.948039 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:21:09.685096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:09.685064 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:10.009012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:10.008979 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:10.153285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:10.153229 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:10.288717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:10.288637 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:12.233392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:12.233362 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:12.365531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:12.365497 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:12.940299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:12.940247 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7"] Oct 31 15:21:13.185074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:13.185039 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:13.307249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:13.307203 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:15.274926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:15.274899 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:15.425225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:15.425198 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:15.555503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:15.555378 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:17.490815 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:21:17.490782 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:21:17.943833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:17.943735 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" podUID="e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" containerName="payload-pre-processing" containerID="cri-o://8712c2a884505f744ea2a3b939754d8ec851491594285b9e081f34e677023890" gracePeriod=25 Oct 31 15:21:18.243916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.243869 4865 generic.go:356] "Generic (PLEG): container finished" podID="e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" containerID="8712c2a884505f744ea2a3b939754d8ec851491594285b9e081f34e677023890" exitCode=0 Oct 31 15:21:18.244128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.243947 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" event={"ID":"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc","Type":"ContainerDied","Data":"8712c2a884505f744ea2a3b939754d8ec851491594285b9e081f34e677023890"} Oct 31 15:21:18.280729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.280706 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:21:18.323020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.322992 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-plugins-config-volume\") pod \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " Oct 31 15:21:18.323149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.323070 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-kube-api-access-7486f\" (UniqueName: \"kubernetes.io/projected/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-kube-api-access-7486f\") pod \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\" (UID: \"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc\") " Oct 31 15:21:18.323349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.323327 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-plugins-config-volume" pod "e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" (UID: "e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:21:18.325225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.325197 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-kube-api-access-7486f" pod "e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" (UID: "e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc"). InnerVolumeSpecName "kube-api-access-7486f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:18.424995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.424963 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7486f\" (UniqueName: \"kubernetes.io/projected/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-kube-api-access-7486f\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:18.424995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:18.424993 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:19.251722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:19.251694 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" Oct 31 15:21:19.252154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:19.251776 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7" event={"ID":"e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc","Type":"ContainerDied","Data":"6dd5d98a2a64e7665c27cf11e6dc66bf7b9f6bdafa0a73ac5c12c8bc3597c9ff"} Oct 31 15:21:19.252154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:19.251826 4865 scope.go:122] "RemoveContainer" containerID="8712c2a884505f744ea2a3b939754d8ec851491594285b9e081f34e677023890" Oct 31 15:21:19.418749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:19.418718 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7"] Oct 31 15:21:19.469998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:19.469965 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9ab5df-6c6b64bb95-2tfr7"] Oct 31 15:21:20.370572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:20.370532 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:20.496605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:20.496569 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" path="/var/lib/kubelet/pods/e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc/volumes" Oct 31 15:21:24.476947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:24.476904 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:25.000061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:25.000025 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:29.575323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:29.575251 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:29.707694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:29.707662 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:29.985796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:29.985742 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:34.514495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:34.500402 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:38.606026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:38.605968 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:39.751156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:39.751115 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:40.103046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.102966 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm"] Oct 31 15:21:40.103298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.103244 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" podUID="26cde22a-296d-4ee4-9c93-d4e4a25aec90" containerName="payload-processing" containerID="cri-o://06d2e1c75f5ca973658e1a88d4530517397e5502170b699e418c536f88feb630" gracePeriod=30 Oct 31 15:21:40.214266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.214226 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht"] Oct 31 15:21:40.214562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.214494 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" podUID="ce936a25-6ca2-420e-9447-71682a664773" containerName="payload-pre-processing" containerID="cri-o://5cbff1fb8acd20c19613a0f666f14376f549b5d571db3d2fa7b47d1c58ed12be" gracePeriod=30 Oct 31 15:21:40.252740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.252704 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:40.385129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.385100 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:40.388675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.388648 4865 generic.go:356] "Generic (PLEG): container finished" podID="ce936a25-6ca2-420e-9447-71682a664773" containerID="5cbff1fb8acd20c19613a0f666f14376f549b5d571db3d2fa7b47d1c58ed12be" exitCode=0 Oct 31 15:21:40.388815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.388755 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" event={"ID":"ce936a25-6ca2-420e-9447-71682a664773","Type":"ContainerDied","Data":"5cbff1fb8acd20c19613a0f666f14376f549b5d571db3d2fa7b47d1c58ed12be"} Oct 31 15:21:40.390521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.390329 4865 generic.go:356] "Generic (PLEG): container finished" podID="26cde22a-296d-4ee4-9c93-d4e4a25aec90" containerID="06d2e1c75f5ca973658e1a88d4530517397e5502170b699e418c536f88feb630" exitCode=0 Oct 31 15:21:40.390521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.390392 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" event={"ID":"26cde22a-296d-4ee4-9c93-d4e4a25aec90","Type":"ContainerDied","Data":"06d2e1c75f5ca973658e1a88d4530517397e5502170b699e418c536f88feb630"} Oct 31 15:21:40.390521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.390437 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" event={"ID":"26cde22a-296d-4ee4-9c93-d4e4a25aec90","Type":"ContainerDied","Data":"680a240df23ff367fc32839c786565524864d49b14abe3d56e41ca296159f0e0"} Oct 31 15:21:40.390521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.390461 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="680a240df23ff367fc32839c786565524864d49b14abe3d56e41ca296159f0e0" Oct 31 15:21:40.412554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.412526 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:21:40.501826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.501800 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:21:40.532191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.531439 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cde22a-296d-4ee4-9c93-d4e4a25aec90-kube-api-access-q6p7w\" (UniqueName: \"kubernetes.io/projected/26cde22a-296d-4ee4-9c93-d4e4a25aec90-kube-api-access-q6p7w\") pod \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " Oct 31 15:21:40.532191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.531678 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cde22a-296d-4ee4-9c93-d4e4a25aec90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cde22a-296d-4ee4-9c93-d4e4a25aec90-plugins-config-volume\") pod \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\" (UID: \"26cde22a-296d-4ee4-9c93-d4e4a25aec90\") " Oct 31 15:21:40.532446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.532194 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cde22a-296d-4ee4-9c93-d4e4a25aec90-plugins-config-volume" pod "26cde22a-296d-4ee4-9c93-d4e4a25aec90" (UID: "26cde22a-296d-4ee4-9c93-d4e4a25aec90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:21:40.533956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.533932 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cde22a-296d-4ee4-9c93-d4e4a25aec90-kube-api-access-q6p7w" pod "26cde22a-296d-4ee4-9c93-d4e4a25aec90" (UID: "26cde22a-296d-4ee4-9c93-d4e4a25aec90"). InnerVolumeSpecName "kube-api-access-q6p7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:40.636252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.633778 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce936a25-6ca2-420e-9447-71682a664773-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce936a25-6ca2-420e-9447-71682a664773-plugins-config-volume\") pod \"ce936a25-6ca2-420e-9447-71682a664773\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " Oct 31 15:21:40.636252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.633891 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce936a25-6ca2-420e-9447-71682a664773-kube-api-access-hrhmk\" (UniqueName: \"kubernetes.io/projected/ce936a25-6ca2-420e-9447-71682a664773-kube-api-access-hrhmk\") pod \"ce936a25-6ca2-420e-9447-71682a664773\" (UID: \"ce936a25-6ca2-420e-9447-71682a664773\") " Oct 31 15:21:40.636252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.634381 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6p7w\" (UniqueName: \"kubernetes.io/projected/26cde22a-296d-4ee4-9c93-d4e4a25aec90-kube-api-access-q6p7w\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:40.636252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.634406 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cde22a-296d-4ee4-9c93-d4e4a25aec90-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:40.660393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.647129 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce936a25-6ca2-420e-9447-71682a664773-kube-api-access-hrhmk" pod "ce936a25-6ca2-420e-9447-71682a664773" (UID: "ce936a25-6ca2-420e-9447-71682a664773"). InnerVolumeSpecName "kube-api-access-hrhmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:40.662076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.662026 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce936a25-6ca2-420e-9447-71682a664773-plugins-config-volume" pod "ce936a25-6ca2-420e-9447-71682a664773" (UID: "ce936a25-6ca2-420e-9447-71682a664773"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:21:40.735440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.735402 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce936a25-6ca2-420e-9447-71682a664773-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:40.735440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:40.735434 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrhmk\" (UniqueName: \"kubernetes.io/projected/ce936a25-6ca2-420e-9447-71682a664773-kube-api-access-hrhmk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:41.398132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.398056 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" Oct 31 15:21:41.398132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.398093 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" event={"ID":"ce936a25-6ca2-420e-9447-71682a664773","Type":"ContainerDied","Data":"9d600c91386563d44960abb6688af2492ca4c7ed811549a71b8086f000949f02"} Oct 31 15:21:41.398736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.398151 4865 scope.go:122] "RemoveContainer" containerID="5cbff1fb8acd20c19613a0f666f14376f549b5d571db3d2fa7b47d1c58ed12be" Oct 31 15:21:41.398736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.398271 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm" Oct 31 15:21:41.443316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.443286 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht" podUID="ce936a25-6ca2-420e-9447-71682a664773" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.195:9004: i/o timeout" Oct 31 15:21:41.565329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.565292 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht"] Oct 31 15:21:41.567974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.567952 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad5bfe-69b74b55cf-nn5ht"] Oct 31 15:21:41.585062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.585029 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:41.690287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.690189 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm"] Oct 31 15:21:41.693517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.693488 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad5bfe-5bdcf59f6b-nsztm"] Oct 31 15:21:41.949229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:41.949156 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:42.044706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.044656 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": read tcp 10.132.0.2:57344->10.132.0.197:15021: read: connection reset by peer" start-of-body= Oct 31 15:21:42.044851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.044726 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": read tcp 10.132.0.2:57344->10.132.0.197:15021: read: connection reset by peer" Oct 31 15:21:42.083367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.083342 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:42.406277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.406237 4865 generic.go:356] "Generic (PLEG): container finished" podID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerID="4e740e2a31bd440e8ec13ae5040fd96b43a8c5bef6ff96d047b945beba15ae6b" exitCode=0 Oct 31 15:21:42.406725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.406312 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" event={"ID":"14345f00-2d2a-4c48-b57f-93e63037a10d","Type":"ContainerDied","Data":"4e740e2a31bd440e8ec13ae5040fd96b43a8c5bef6ff96d047b945beba15ae6b"} Oct 31 15:21:42.406780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.406735 4865 scope.go:122] "RemoveContainer" containerID="4e740e2a31bd440e8ec13ae5040fd96b43a8c5bef6ff96d047b945beba15ae6b" Oct 31 15:21:42.408271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.408239 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:21:42.408407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.408319 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:21:42.408407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.408369 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:21:42.498333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.498306 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cde22a-296d-4ee4-9c93-d4e4a25aec90" path="/var/lib/kubelet/pods/26cde22a-296d-4ee4-9c93-d4e4a25aec90/volumes" Oct 31 15:21:42.498814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:42.498797 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce936a25-6ca2-420e-9447-71682a664773" path="/var/lib/kubelet/pods/ce936a25-6ca2-420e-9447-71682a664773/volumes" Oct 31 15:21:43.426050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:43.426013 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" event={"ID":"14345f00-2d2a-4c48-b57f-93e63037a10d","Type":"ContainerStarted","Data":"a8f9f1365cb3ed30eee6958db0f467dd27348d75ea2ad4cd24934c194fc29b7a"} Oct 31 15:21:43.518545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:43.505009 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:43.974341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:43.974299 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt"] Oct 31 15:21:43.974599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:43.974574 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" containerID="cri-o://f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151" gracePeriod=30 Oct 31 15:21:44.018381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:44.018354 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:21:44.020584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:44.019031 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 15:21:44.020790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:44.020767 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 15:21:44.024116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:44.024087 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:45.014494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:45.010899 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 15:21:45.014494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:45.010963 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 15:21:45.521186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:45.521154 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:46.009500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.008784 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 15:21:46.009500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.008854 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 15:21:46.040037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040008 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55"] Oct 31 15:21:46.040705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040689 4865 cpu_manager.go:414] "containerMap: removing container" podUID="171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" containerName="payload-pre-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040707 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" containerName="payload-pre-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040723 4865 cpu_manager.go:414] "containerMap: removing container" podUID="26cde22a-296d-4ee4-9c93-d4e4a25aec90" containerName="payload-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040728 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cde22a-296d-4ee4-9c93-d4e4a25aec90" containerName="payload-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040747 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7a20fe9f-e754-480f-b136-70ce4beee1c3" containerName="payload-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040754 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a20fe9f-e754-480f-b136-70ce4beee1c3" containerName="payload-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040770 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" containerName="payload-pre-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040776 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" containerName="payload-pre-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040793 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ce936a25-6ca2-420e-9447-71682a664773" containerName="payload-pre-processing" Oct 31 15:21:46.040824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040800 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce936a25-6ca2-420e-9447-71682a664773" containerName="payload-pre-processing" Oct 31 15:21:46.041415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040931 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a20fe9f-e754-480f-b136-70ce4beee1c3" containerName="payload-processing" Oct 31 15:21:46.041415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040944 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2178e3f-4e0f-4bae-ae17-da74a5b9ccfc" containerName="payload-pre-processing" Oct 31 15:21:46.041415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040953 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce936a25-6ca2-420e-9447-71682a664773" containerName="payload-pre-processing" Oct 31 15:21:46.041415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040964 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="171ca979-9d95-4ead-b4a6-f6a5ff8e14b9" containerName="payload-pre-processing" Oct 31 15:21:46.041415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.040970 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cde22a-296d-4ee4-9c93-d4e4a25aec90" containerName="payload-processing" Oct 31 15:21:46.045665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.045644 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.127259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.127226 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b3065a-openshift-default-dockercfg-n2gdt\"" Oct 31 15:21:46.128398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.128238 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:46.188501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.179245 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-certs\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.188501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.179305 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-podinfo\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.188501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.179350 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istiod-ca-cert\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.188501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.179387 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-credential-socket\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.245061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.245024 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-token\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.250631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.250589 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-envoy\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.250911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.250888 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-socket\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.251118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.251101 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-data\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.251269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.251252 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46599\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-kube-api-access-46599\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.287995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.287911 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55"] Oct 31 15:21:46.352342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352306 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-token\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.352566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352348 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-envoy\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.352566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352535 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-socket\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.352700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352631 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-data\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.352700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352682 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46599\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-kube-api-access-46599\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.352823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352802 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-certs\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.352890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352849 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-podinfo\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.352970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352943 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-socket\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.353036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352966 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-data\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.353036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.352972 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istiod-ca-cert\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.353036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.353026 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-credential-socket\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.353210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.353100 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-certs\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.353381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.353363 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-credential-socket\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.353679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.353659 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istiod-ca-cert\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.354932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.354911 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-envoy\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.355384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.355366 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-podinfo\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.419580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.419549 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46599\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-kube-api-access-46599\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.420617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.420594 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-token\") pod \"e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.601555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.599541 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:46.660875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.660842 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:46.868078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:46.867982 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:47.019516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.007576 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 15:21:47.019516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.007633 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 15:21:47.019516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.011566 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:47.123341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.123310 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55"] Oct 31 15:21:47.124702 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:21:47.124675 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f26fd5_25d9_41c0_bd46_b6d03361bfe1.slice/crio-9ca2b9df95ea044bf7490ae783a790812dd1dccaf2cff384280471ddf9623ad2 WatchSource:0}: Error finding container 9ca2b9df95ea044bf7490ae783a790812dd1dccaf2cff384280471ddf9623ad2: Status 404 returned error can't find the container with id 9ca2b9df95ea044bf7490ae783a790812dd1dccaf2cff384280471ddf9623ad2 Oct 31 15:21:47.127266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.127236 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:21:47.127354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.127304 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:21:47.127354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.127336 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:21:47.455313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.455221 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" event={"ID":"69f26fd5-25d9-41c0-bd46-b6d03361bfe1","Type":"ContainerStarted","Data":"173c81ec60901bb2bcd39d25a2012947e37757f69a8cf6b5396f139a4a3b4734"} Oct 31 15:21:47.455313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.455282 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" event={"ID":"69f26fd5-25d9-41c0-bd46-b6d03361bfe1","Type":"ContainerStarted","Data":"9ca2b9df95ea044bf7490ae783a790812dd1dccaf2cff384280471ddf9623ad2"} Oct 31 15:21:47.491629 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:21:47.491592 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:21:47.585589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.585529 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podStartSLOduration=2.58550913 podStartE2EDuration="2.58550913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:21:47.580899091 +0000 UTC m=+2153.200450542" watchObservedRunningTime="2026-10-31 15:21:47.58550913 +0000 UTC m=+2153.205060540" Oct 31 15:21:47.661250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:47.661220 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:48.013495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:48.009814 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 15:21:48.013495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:48.009875 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 15:21:48.437135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:48.437049 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:48.667080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:48.667042 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:48.931929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:48.931893 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:49.062918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.057744 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:21:49.062918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.059229 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:21:49.062918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.061416 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:21:49.251410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.251384 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:21:49.417748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.417714 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-data\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.417908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.417795 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-kube-api-access-wzxgz\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-kube-api-access-wzxgz\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.417908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.417829 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-envoy\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.417908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.417900 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-token\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-token\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.418060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.417936 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-credential-socket\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.418060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.417988 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-podinfo\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.418060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.418020 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-socket\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.418159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.418109 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istiod-ca-cert\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.418159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.418144 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-certs\") pod \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\" (UID: \"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f\") " Oct 31 15:21:49.419321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.418869 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-credential-socket" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:21:49.419321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.418881 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-certs" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:21:49.419321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.418897 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-data" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:21:49.419321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.419129 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istiod-ca-cert" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:21:49.419321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.419295 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-socket" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:21:49.421379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.421349 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-envoy" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:21:49.421524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.421377 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-kube-api-access-wzxgz" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "kube-api-access-wzxgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:49.424837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.424815 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-token" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:49.426619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.426593 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-podinfo" pod "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" (UID: "2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:21:49.470325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.470292 4865 generic.go:356] "Generic (PLEG): container finished" podID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerID="f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151" exitCode=0 Oct 31 15:21:49.470796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.470355 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" Oct 31 15:21:49.470796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.470369 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" event={"ID":"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f","Type":"ContainerDied","Data":"f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151"} Oct 31 15:21:49.470796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.470418 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt" event={"ID":"2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f","Type":"ContainerDied","Data":"7e8bd06da791bc287393a33429c09219419fbe04b28e2d17a3e01e4ebde27d7d"} Oct 31 15:21:49.470796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.470438 4865 scope.go:122] "RemoveContainer" containerID="f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151" Oct 31 15:21:49.471653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.471041 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:49.472010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.471990 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:21:49.480869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.480852 4865 scope.go:122] "RemoveContainer" containerID="a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d" Oct 31 15:21:49.491900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.491880 4865 scope.go:122] "RemoveContainer" containerID="f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151" Oct 31 15:21:49.492204 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:21:49.492187 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151\": container with ID starting with f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151 not found: ID does not exist" containerID="f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151" Oct 31 15:21:49.492274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.492213 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151"} err="failed to get container status \"f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151\": rpc error: code = NotFound desc = could not find container \"f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151\": container with ID starting with f1acb4f1679a8354b71f85875ea85d9332b50180d05074e379dee2789bb10151 not found: ID does not exist" Oct 31 15:21:49.492274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.492231 4865 scope.go:122] "RemoveContainer" containerID="a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d" Oct 31 15:21:49.492486 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:21:49.492456 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d\": container with ID starting with a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d not found: ID does not exist" containerID="a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d" Oct 31 15:21:49.492546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.492489 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d"} err="failed to get container status \"a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d\": rpc error: code = NotFound desc = could not find container \"a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d\": container with ID starting with a321e27c3ffabd1a712e7e32b844315d9c1348f20284984c23b7cd190b840a0d not found: ID does not exist" Oct 31 15:21:49.520001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.519972 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.519997 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.520008 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.520017 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzxgz\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-kube-api-access-wzxgz\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.520026 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.520035 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.520043 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.520051 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.520159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.520060 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:49.714688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.714659 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:49.760586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.760501 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt"] Oct 31 15:21:49.764785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:49.764761 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad5bfe-openshift-default-6b4b6c5567-rmcgt"] Oct 31 15:21:50.251787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:50.251744 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:21:50.336688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:50.336661 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:50.499947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:50.499911 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" path="/var/lib/kubelet/pods/2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f/volumes" Oct 31 15:21:50.524801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:50.524722 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:50.652390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:50.652355 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:51.914913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:51.914875 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:53.619852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:53.619824 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:53.839495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:53.817376 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:54.207494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.199864 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs"] Oct 31 15:21:54.207494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.200827 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" Oct 31 15:21:54.207494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.200848 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" Oct 31 15:21:54.207494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.200868 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" Oct 31 15:21:54.207494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.200874 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" Oct 31 15:21:54.207494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.201010 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" Oct 31 15:21:54.207494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.201026 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2519b6be-d5ff-4d21-a934-f4b9f5ca1b9f" containerName="istio-proxy" Oct 31 15:21:54.225500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.223444 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.272288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.272252 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs"] Oct 31 15:21:54.310492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.306721 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b3065a\"" Oct 31 15:21:54.310492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.306869 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b3065a-dockercfg-xnmzk\"" Oct 31 15:21:54.358264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.358218 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20346d1-6975-4d7e-b7f7-240b262139cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.358542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.358282 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ck5s\" (UniqueName: \"kubernetes.io/projected/c20346d1-6975-4d7e-b7f7-240b262139cf-kube-api-access-8ck5s\") pod \"payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.361179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.361154 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8"] Oct 31 15:21:54.365350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.365331 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:54.423094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.423061 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8"] Oct 31 15:21:54.460178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.460094 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20346d1-6975-4d7e-b7f7-240b262139cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.460178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.460154 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ck5s\" (UniqueName: \"kubernetes.io/projected/c20346d1-6975-4d7e-b7f7-240b262139cf-kube-api-access-8ck5s\") pod \"payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.460392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.460187 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d646022-1a55-477a-9242-a3db284260bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:54.460392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.460256 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9pk\" (UniqueName: \"kubernetes.io/projected/5d646022-1a55-477a-9242-a3db284260bd-kube-api-access-rb9pk\") pod \"payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:54.460924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.460893 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20346d1-6975-4d7e-b7f7-240b262139cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.523629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.523554 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ck5s\" (UniqueName: \"kubernetes.io/projected/c20346d1-6975-4d7e-b7f7-240b262139cf-kube-api-access-8ck5s\") pod \"payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.561865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.561833 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9pk\" (UniqueName: \"kubernetes.io/projected/5d646022-1a55-477a-9242-a3db284260bd-kube-api-access-rb9pk\") pod \"payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:54.562059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.562006 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d646022-1a55-477a-9242-a3db284260bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:54.562705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.562684 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d646022-1a55-477a-9242-a3db284260bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:54.621421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.621383 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9pk\" (UniqueName: \"kubernetes.io/projected/5d646022-1a55-477a-9242-a3db284260bd-kube-api-access-rb9pk\") pod \"payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:54.644849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.644812 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b3065a-dockercfg-xnmzk\"" Oct 31 15:21:54.652942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.652905 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:54.678797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:54.678774 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:55.054623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.054601 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs"] Oct 31 15:21:55.054786 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:21:55.054760 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20346d1_6975_4d7e_b7f7_240b262139cf.slice/crio-f87a2ca4bcadade0dbb7555e5fdae1c736cde7cb7b3b51c5de58ac452a5911e8 WatchSource:0}: Error finding container f87a2ca4bcadade0dbb7555e5fdae1c736cde7cb7b3b51c5de58ac452a5911e8: Status 404 returned error can't find the container with id f87a2ca4bcadade0dbb7555e5fdae1c736cde7cb7b3b51c5de58ac452a5911e8 Oct 31 15:21:55.123392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.123361 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8"] Oct 31 15:21:55.133553 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:21:55.133508 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d646022_1a55_477a_9242_a3db284260bd.slice/crio-c201a18ad4b95e33485c3057b710fedc0ccbc63fea954f4249c92d565a76475c WatchSource:0}: Error finding container c201a18ad4b95e33485c3057b710fedc0ccbc63fea954f4249c92d565a76475c: Status 404 returned error can't find the container with id c201a18ad4b95e33485c3057b710fedc0ccbc63fea954f4249c92d565a76475c Oct 31 15:21:55.542857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.542820 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" event={"ID":"5d646022-1a55-477a-9242-a3db284260bd","Type":"ContainerStarted","Data":"b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770"} Oct 31 15:21:55.543064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.542883 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" event={"ID":"5d646022-1a55-477a-9242-a3db284260bd","Type":"ContainerStarted","Data":"c201a18ad4b95e33485c3057b710fedc0ccbc63fea954f4249c92d565a76475c"} Oct 31 15:21:55.543064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.543010 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:21:55.544410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.544386 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" event={"ID":"c20346d1-6975-4d7e-b7f7-240b262139cf","Type":"ContainerStarted","Data":"51a6fff52fef94803fd1c83f3d43137272afa6f46a67e46a92274ed949093566"} Oct 31 15:21:55.544509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.544423 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" event={"ID":"c20346d1-6975-4d7e-b7f7-240b262139cf","Type":"ContainerStarted","Data":"f87a2ca4bcadade0dbb7555e5fdae1c736cde7cb7b3b51c5de58ac452a5911e8"} Oct 31 15:21:55.544509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.544491 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:21:55.612497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.610876 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:55.672085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.672032 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" podStartSLOduration=1.672017251 podStartE2EDuration="1.672017251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:21:55.671304634 +0000 UTC m=+2161.290856236" watchObservedRunningTime="2026-10-31 15:21:55.672017251 +0000 UTC m=+2161.291568661" Oct 31 15:21:55.807543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:55.802267 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" podStartSLOduration=1.80223538 podStartE2EDuration="1.80223538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:21:55.797963143 +0000 UTC m=+2161.417514565" watchObservedRunningTime="2026-10-31 15:21:55.80223538 +0000 UTC m=+2161.421786786" Oct 31 15:21:56.950324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:56.950292 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:57.077908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:57.077878 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq"] Oct 31 15:21:57.189107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:57.189018 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl"] Oct 31 15:21:57.384719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:57.384683 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:57.555131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:57.555014 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:57.681376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:57.681349 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:21:58.342715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:21:58.342681 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:01.045246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.044353 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": read tcp 10.132.0.2:36752->10.132.0.198:15021: read: connection reset by peer" start-of-body= Oct 31 15:22:01.045246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.044420 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": read tcp 10.132.0.2:36752->10.132.0.198:15021: read: connection reset by peer" Oct 31 15:22:01.438185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.438095 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:01.592556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.592520 4865 generic.go:356] "Generic (PLEG): container finished" podID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerID="e28b2fd24e44a296f5bf2ae4f7863d0f13e872ec7733327d4686a6625af67cd0" exitCode=0 Oct 31 15:22:01.592761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.592598 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" event={"ID":"539e7c13-c6e7-49ad-a03d-617a87a5157b","Type":"ContainerDied","Data":"e28b2fd24e44a296f5bf2ae4f7863d0f13e872ec7733327d4686a6625af67cd0"} Oct 31 15:22:01.593107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.593086 4865 scope.go:122] "RemoveContainer" containerID="e28b2fd24e44a296f5bf2ae4f7863d0f13e872ec7733327d4686a6625af67cd0" Oct 31 15:22:01.594692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.594660 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:22:01.594794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.594748 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:22:01.594866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:01.594801 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:22:02.078633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.078567 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" podUID="33df4b69-5198-4a06-8df2-8ea612beca27" containerName="payload-processing" containerID="cri-o://9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28" gracePeriod=25 Oct 31 15:22:02.189655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.189609 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" podUID="d6e233da-d5cd-41d0-8b04-af6edc905ca8" containerName="payload-pre-processing" containerID="cri-o://8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903" gracePeriod=25 Oct 31 15:22:02.432492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.426435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:02.447510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.447294 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:22:02.536031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.535991 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33df4b69-5198-4a06-8df2-8ea612beca27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df4b69-5198-4a06-8df2-8ea612beca27-plugins-config-volume\") pod \"33df4b69-5198-4a06-8df2-8ea612beca27\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " Oct 31 15:22:02.536192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.536102 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33df4b69-5198-4a06-8df2-8ea612beca27-kube-api-access-sh9f7\" (UniqueName: \"kubernetes.io/projected/33df4b69-5198-4a06-8df2-8ea612beca27-kube-api-access-sh9f7\") pod \"33df4b69-5198-4a06-8df2-8ea612beca27\" (UID: \"33df4b69-5198-4a06-8df2-8ea612beca27\") " Oct 31 15:22:02.537226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.537193 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33df4b69-5198-4a06-8df2-8ea612beca27-plugins-config-volume" pod "33df4b69-5198-4a06-8df2-8ea612beca27" (UID: "33df4b69-5198-4a06-8df2-8ea612beca27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:02.539433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.539368 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33df4b69-5198-4a06-8df2-8ea612beca27-kube-api-access-sh9f7" pod "33df4b69-5198-4a06-8df2-8ea612beca27" (UID: "33df4b69-5198-4a06-8df2-8ea612beca27"). InnerVolumeSpecName "kube-api-access-sh9f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:02.588837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.588771 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:22:02.630875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.630840 4865 generic.go:356] "Generic (PLEG): container finished" podID="33df4b69-5198-4a06-8df2-8ea612beca27" containerID="9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28" exitCode=0 Oct 31 15:22:02.631050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.630890 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" Oct 31 15:22:02.631050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.630921 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" event={"ID":"33df4b69-5198-4a06-8df2-8ea612beca27","Type":"ContainerDied","Data":"9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28"} Oct 31 15:22:02.631050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.630980 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq" event={"ID":"33df4b69-5198-4a06-8df2-8ea612beca27","Type":"ContainerDied","Data":"55048367b9f3d8d24df3da1894c9485918a176656b7a7c9942951d263e78816c"} Oct 31 15:22:02.631050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.631001 4865 scope.go:122] "RemoveContainer" containerID="9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28" Oct 31 15:22:02.633640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.633619 4865 generic.go:356] "Generic (PLEG): container finished" podID="d6e233da-d5cd-41d0-8b04-af6edc905ca8" containerID="8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903" exitCode=0 Oct 31 15:22:02.634136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.633967 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" Oct 31 15:22:02.634235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.634162 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" event={"ID":"d6e233da-d5cd-41d0-8b04-af6edc905ca8","Type":"ContainerDied","Data":"8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903"} Oct 31 15:22:02.634235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.634203 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl" event={"ID":"d6e233da-d5cd-41d0-8b04-af6edc905ca8","Type":"ContainerDied","Data":"85782ee7a075731f5e43a44cd90c9ac35e6d06068daa7392b6ca025318ac4948"} Oct 31 15:22:02.641504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.636642 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" event={"ID":"539e7c13-c6e7-49ad-a03d-617a87a5157b","Type":"ContainerStarted","Data":"5d7dcda2bbdfa389fee9589816ecd838251433efc1f131dd071469189127664a"} Oct 31 15:22:02.641504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.637412 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh9f7\" (UniqueName: \"kubernetes.io/projected/33df4b69-5198-4a06-8df2-8ea612beca27-kube-api-access-sh9f7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:02.641504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.637437 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df4b69-5198-4a06-8df2-8ea612beca27-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:02.642980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.642956 4865 scope.go:122] "RemoveContainer" containerID="9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28" Oct 31 15:22:02.643259 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:22:02.643237 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28\": container with ID starting with 9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28 not found: ID does not exist" containerID="9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28" Oct 31 15:22:02.643324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.643265 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28"} err="failed to get container status \"9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28\": rpc error: code = NotFound desc = could not find container \"9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28\": container with ID starting with 9c52ee8e5d74744b3a38f27538783782330ada7003d8836070c83b4eb2c48c28 not found: ID does not exist" Oct 31 15:22:02.643324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.643282 4865 scope.go:122] "RemoveContainer" containerID="8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903" Oct 31 15:22:02.652142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.652125 4865 scope.go:122] "RemoveContainer" containerID="8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903" Oct 31 15:22:02.652411 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:22:02.652393 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903\": container with ID starting with 8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903 not found: ID does not exist" containerID="8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903" Oct 31 15:22:02.652478 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.652421 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903"} err="failed to get container status \"8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903\": rpc error: code = NotFound desc = could not find container \"8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903\": container with ID starting with 8d052b8d19f54349b5d7614d6bbeb21e29582405cd1fd550862a01497eeb0903 not found: ID does not exist" Oct 31 15:22:02.737842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.737811 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e233da-d5cd-41d0-8b04-af6edc905ca8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e233da-d5cd-41d0-8b04-af6edc905ca8-plugins-config-volume\") pod \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " Oct 31 15:22:02.738209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.738193 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e233da-d5cd-41d0-8b04-af6edc905ca8-kube-api-access-lb9xj\" (UniqueName: \"kubernetes.io/projected/d6e233da-d5cd-41d0-8b04-af6edc905ca8-kube-api-access-lb9xj\") pod \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\" (UID: \"d6e233da-d5cd-41d0-8b04-af6edc905ca8\") " Oct 31 15:22:02.738333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.738265 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e233da-d5cd-41d0-8b04-af6edc905ca8-plugins-config-volume" pod "d6e233da-d5cd-41d0-8b04-af6edc905ca8" (UID: "d6e233da-d5cd-41d0-8b04-af6edc905ca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:02.738827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.738795 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6e233da-d5cd-41d0-8b04-af6edc905ca8-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:02.740603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.740578 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e233da-d5cd-41d0-8b04-af6edc905ca8-kube-api-access-lb9xj" pod "d6e233da-d5cd-41d0-8b04-af6edc905ca8" (UID: "d6e233da-d5cd-41d0-8b04-af6edc905ca8"). InnerVolumeSpecName "kube-api-access-lb9xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:02.839744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.839656 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb9xj\" (UniqueName: \"kubernetes.io/projected/d6e233da-d5cd-41d0-8b04-af6edc905ca8-kube-api-access-lb9xj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:02.951974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.951947 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq"] Oct 31 15:22:02.957110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:02.957087 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b62069-85dccd448c-l78cq"] Oct 31 15:22:03.132369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:03.132337 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl"] Oct 31 15:22:03.135718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:03.135696 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b62069-78b9d6b95f-g4cxl"] Oct 31 15:22:03.462080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:03.462051 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:22:03.462256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:03.462231 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 15:22:03.462313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:03.462293 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 15:22:04.462252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:04.462216 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 15:22:04.462677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:04.462284 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 15:22:04.520895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:04.520865 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33df4b69-5198-4a06-8df2-8ea612beca27" path="/var/lib/kubelet/pods/33df4b69-5198-4a06-8df2-8ea612beca27/volumes" Oct 31 15:22:04.521259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:04.521245 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e233da-d5cd-41d0-8b04-af6edc905ca8" path="/var/lib/kubelet/pods/d6e233da-d5cd-41d0-8b04-af6edc905ca8/volumes" Oct 31 15:22:05.462384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:05.462346 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 15:22:05.462837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:05.462409 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 15:22:06.462494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:06.462326 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 15:22:06.462494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:06.462382 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 15:22:06.553876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:06.553847 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:22:06.554029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:06.553921 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:22:07.462030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:07.461992 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 31 15:22:07.462225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:07.462049 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 31 15:22:07.560072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:07.560032 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:08.466063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:08.466034 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:22:08.466491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:08.466446 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:22:08.467183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:08.467162 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:22:12.684100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:12.684067 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:15.771794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:15.771755 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:16.890576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:16.890545 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:17.413524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.413484 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk"] Oct 31 15:22:17.414044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.414031 4865 cpu_manager.go:414] "containerMap: removing container" podUID="33df4b69-5198-4a06-8df2-8ea612beca27" containerName="payload-processing" Oct 31 15:22:17.414102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.414045 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33df4b69-5198-4a06-8df2-8ea612beca27" containerName="payload-processing" Oct 31 15:22:17.414102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.414085 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d6e233da-d5cd-41d0-8b04-af6edc905ca8" containerName="payload-pre-processing" Oct 31 15:22:17.414102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.414090 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e233da-d5cd-41d0-8b04-af6edc905ca8" containerName="payload-pre-processing" Oct 31 15:22:17.414224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.414177 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e233da-d5cd-41d0-8b04-af6edc905ca8" containerName="payload-pre-processing" Oct 31 15:22:17.414224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.414195 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="33df4b69-5198-4a06-8df2-8ea612beca27" containerName="payload-processing" Oct 31 15:22:17.418750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.418732 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.483494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.474533 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ebadea-openshift-default-dockercfg-5wp6j\"" Oct 31 15:22:17.552494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552023 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489b6427-e570-48d5-8ded-d0065ffc6393-istio-podinfo\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552091 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-socket\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552137 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghl4\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-kube-api-access-dghl4\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-istio-token\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552267 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-certs\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552366 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-credential-socket\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552418 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489b6427-e570-48d5-8ded-d0065ffc6393-istiod-ca-cert\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552518 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-envoy\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.552957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.552553 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-data\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.582197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.582164 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk"] Oct 31 15:22:17.617360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.617063 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:17.619518 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:22:17.618217 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:22:17.653839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.653804 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-certs\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.653902 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-credential-socket\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.653933 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489b6427-e570-48d5-8ded-d0065ffc6393-istiod-ca-cert\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.653976 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-envoy\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654004 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-data\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654047 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489b6427-e570-48d5-8ded-d0065ffc6393-istio-podinfo\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654070 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-socket\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654111 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghl4\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-kube-api-access-dghl4\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654168 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-istio-token\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654192 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-certs\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654328 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-credential-socket\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654533 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-socket\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654695 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-data\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.654726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.654702 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489b6427-e570-48d5-8ded-d0065ffc6393-istiod-ca-cert\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.656586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.656567 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-envoy\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.656702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.656646 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489b6427-e570-48d5-8ded-d0065ffc6393-istio-podinfo\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.714380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.714343 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-istio-token\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.714678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.714484 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghl4\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-kube-api-access-dghl4\") pod \"e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.756446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.756412 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:17.856690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.856643 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:17.998997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:17.998902 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:18.225279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.225251 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk"] Oct 31 15:22:18.225592 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:22:18.225570 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489b6427_e570_48d5_8ded_d0065ffc6393.slice/crio-5ea67681c0029b0d32c3d2e066fffe884abc3b063def33ae38736e709733b615 WatchSource:0}: Error finding container 5ea67681c0029b0d32c3d2e066fffe884abc3b063def33ae38736e709733b615: Status 404 returned error can't find the container with id 5ea67681c0029b0d32c3d2e066fffe884abc3b063def33ae38736e709733b615 Oct 31 15:22:18.228119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.228076 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:22:18.228185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.228155 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:22:18.228227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.228191 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:22:18.746266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.746222 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" event={"ID":"489b6427-e570-48d5-8ded-d0065ffc6393","Type":"ContainerStarted","Data":"82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01"} Oct 31 15:22:18.746535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.746295 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" event={"ID":"489b6427-e570-48d5-8ded-d0065ffc6393","Type":"ContainerStarted","Data":"5ea67681c0029b0d32c3d2e066fffe884abc3b063def33ae38736e709733b615"} Oct 31 15:22:18.759551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.759518 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:18.871518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:18.871451 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" podStartSLOduration=1.871438186 podStartE2EDuration="1.871438186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:22:18.869911921 +0000 UTC m=+2184.489463333" watchObservedRunningTime="2026-10-31 15:22:18.871438186 +0000 UTC m=+2184.490989597" Oct 31 15:22:19.762196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:19.762171 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:20.303143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:20.303111 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:20.760857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:20.760822 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:20.761722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:20.761705 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:22:21.645427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:21.645384 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:21.824707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:21.824672 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:21.968078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:21.968044 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:22.393736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:22.393654 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:23.231048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:23.231001 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:24.889201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:24.889162 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:25.060726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.060684 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:25.307558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.307530 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247"] Oct 31 15:22:25.311714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.311699 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.358478 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.358450 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6329fe9-80df-4026-8628-ccc4468eb8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.358627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.358585 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z522d\" (UniqueName: \"kubernetes.io/projected/a6329fe9-80df-4026-8628-ccc4468eb8e1-kube-api-access-z522d\") pod \"payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.369601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.369529 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247"] Oct 31 15:22:25.416539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.416502 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ebadea\"" Oct 31 15:22:25.466393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.466350 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6329fe9-80df-4026-8628-ccc4468eb8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.471005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.470975 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z522d\" (UniqueName: \"kubernetes.io/projected/a6329fe9-80df-4026-8628-ccc4468eb8e1-kube-api-access-z522d\") pod \"payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.471902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.471738 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6329fe9-80df-4026-8628-ccc4468eb8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.511570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.511535 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz"] Oct 31 15:22:25.525292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.525261 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:25.535079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.535044 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz"] Oct 31 15:22:25.539147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.539123 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z522d\" (UniqueName: \"kubernetes.io/projected/a6329fe9-80df-4026-8628-ccc4468eb8e1-kube-api-access-z522d\") pod \"payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.573460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.573381 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:25.573640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.573538 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djwjc\" (UniqueName: \"kubernetes.io/projected/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-kube-api-access-djwjc\") pod \"payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:25.579670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.579643 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ebadea-dockercfg-xzwhr\"" Oct 31 15:22:25.626264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.626232 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:25.675356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.675320 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:25.675573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.675386 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djwjc\" (UniqueName: \"kubernetes.io/projected/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-kube-api-access-djwjc\") pod \"payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:25.676103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.676083 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:25.734756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.734721 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djwjc\" (UniqueName: \"kubernetes.io/projected/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-kube-api-access-djwjc\") pod \"payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:25.862798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:25.862711 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:26.011917 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:22:26.011883 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6329fe9_80df_4026_8628_ccc4468eb8e1.slice/crio-4c3678501fb481908a00c4719433a53d3ea31715e6c48027ad715fc0a6d7aac2 WatchSource:0}: Error finding container 4c3678501fb481908a00c4719433a53d3ea31715e6c48027ad715fc0a6d7aac2: Status 404 returned error can't find the container with id 4c3678501fb481908a00c4719433a53d3ea31715e6c48027ad715fc0a6d7aac2 Oct 31 15:22:26.012303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.011994 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247"] Oct 31 15:22:26.239142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.239113 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz"] Oct 31 15:22:26.239709 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:22:26.239686 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd1fe3f_3174_400f_b20c_ab3fca64ed88.slice/crio-808c255024d4937e262cc92281a750e615e2e458cabf15b02f305b5a97f06054 WatchSource:0}: Error finding container 808c255024d4937e262cc92281a750e615e2e458cabf15b02f305b5a97f06054: Status 404 returned error can't find the container with id 808c255024d4937e262cc92281a750e615e2e458cabf15b02f305b5a97f06054 Oct 31 15:22:26.804143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.804092 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" event={"ID":"0fd1fe3f-3174-400f-b20c-ab3fca64ed88","Type":"ContainerStarted","Data":"917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253"} Oct 31 15:22:26.804373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.804178 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" event={"ID":"0fd1fe3f-3174-400f-b20c-ab3fca64ed88","Type":"ContainerStarted","Data":"808c255024d4937e262cc92281a750e615e2e458cabf15b02f305b5a97f06054"} Oct 31 15:22:26.804373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.804308 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:26.805878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.805850 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" event={"ID":"a6329fe9-80df-4026-8628-ccc4468eb8e1","Type":"ContainerStarted","Data":"d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7"} Oct 31 15:22:26.806005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.805889 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" event={"ID":"a6329fe9-80df-4026-8628-ccc4468eb8e1","Type":"ContainerStarted","Data":"4c3678501fb481908a00c4719433a53d3ea31715e6c48027ad715fc0a6d7aac2"} Oct 31 15:22:26.806005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.805910 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:26.921432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:26.921382 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" podStartSLOduration=1.921368465 podStartE2EDuration="1.921368465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:22:26.919393691 +0000 UTC m=+2192.538945098" watchObservedRunningTime="2026-10-31 15:22:26.921368465 +0000 UTC m=+2192.540919876" Oct 31 15:22:27.058542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:27.045981 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" podStartSLOduration=2.045961629 podStartE2EDuration="2.045961629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:22:27.040950348 +0000 UTC m=+2192.660501760" watchObservedRunningTime="2026-10-31 15:22:27.045961629 +0000 UTC m=+2192.665513041" Oct 31 15:22:28.109988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:28.109954 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:28.338119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:28.338088 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm"] Oct 31 15:22:28.450715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:28.450626 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7"] Oct 31 15:22:28.547743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:28.547690 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:28.694607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:28.694574 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:28.826288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:28.826260 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:33.338979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.338935 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" podUID="c8505626-7c66-4416-90c5-177d9ca407e5" containerName="payload-processing" containerID="cri-o://632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937" gracePeriod=25 Oct 31 15:22:33.451535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.451449 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" podUID="189fd5aa-4825-426d-bfa3-f601a4aaa71b" containerName="payload-pre-processing" containerID="cri-o://a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc" gracePeriod=25 Oct 31 15:22:33.548565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.548458 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:33.624908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.624876 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:22:33.709679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.709651 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8505626-7c66-4416-90c5-177d9ca407e5-kube-api-access-fjdmg\" (UniqueName: \"kubernetes.io/projected/c8505626-7c66-4416-90c5-177d9ca407e5-kube-api-access-fjdmg\") pod \"c8505626-7c66-4416-90c5-177d9ca407e5\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " Oct 31 15:22:33.709679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.709683 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8505626-7c66-4416-90c5-177d9ca407e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8505626-7c66-4416-90c5-177d9ca407e5-plugins-config-volume\") pod \"c8505626-7c66-4416-90c5-177d9ca407e5\" (UID: \"c8505626-7c66-4416-90c5-177d9ca407e5\") " Oct 31 15:22:33.710108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.710079 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8505626-7c66-4416-90c5-177d9ca407e5-plugins-config-volume" pod "c8505626-7c66-4416-90c5-177d9ca407e5" (UID: "c8505626-7c66-4416-90c5-177d9ca407e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:33.712046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.712023 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8505626-7c66-4416-90c5-177d9ca407e5-kube-api-access-fjdmg" pod "c8505626-7c66-4416-90c5-177d9ca407e5" (UID: "c8505626-7c66-4416-90c5-177d9ca407e5"). InnerVolumeSpecName "kube-api-access-fjdmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:33.732771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.732752 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:22:33.744966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.744940 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:33.811084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.811049 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189fd5aa-4825-426d-bfa3-f601a4aaa71b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189fd5aa-4825-426d-bfa3-f601a4aaa71b-plugins-config-volume\") pod \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " Oct 31 15:22:33.811303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.811139 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189fd5aa-4825-426d-bfa3-f601a4aaa71b-kube-api-access-7nlwm\" (UniqueName: \"kubernetes.io/projected/189fd5aa-4825-426d-bfa3-f601a4aaa71b-kube-api-access-7nlwm\") pod \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\" (UID: \"189fd5aa-4825-426d-bfa3-f601a4aaa71b\") " Oct 31 15:22:33.811568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.811540 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189fd5aa-4825-426d-bfa3-f601a4aaa71b-plugins-config-volume" pod "189fd5aa-4825-426d-bfa3-f601a4aaa71b" (UID: "189fd5aa-4825-426d-bfa3-f601a4aaa71b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:33.811657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.811639 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjdmg\" (UniqueName: \"kubernetes.io/projected/c8505626-7c66-4416-90c5-177d9ca407e5-kube-api-access-fjdmg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:33.811657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.811654 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8505626-7c66-4416-90c5-177d9ca407e5-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:33.811790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.811663 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189fd5aa-4825-426d-bfa3-f601a4aaa71b-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:33.813909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.813885 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189fd5aa-4825-426d-bfa3-f601a4aaa71b-kube-api-access-7nlwm" pod "189fd5aa-4825-426d-bfa3-f601a4aaa71b" (UID: "189fd5aa-4825-426d-bfa3-f601a4aaa71b"). InnerVolumeSpecName "kube-api-access-7nlwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:33.852878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.852808 4865 generic.go:356] "Generic (PLEG): container finished" podID="189fd5aa-4825-426d-bfa3-f601a4aaa71b" containerID="a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc" exitCode=0 Oct 31 15:22:33.852983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.852874 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" Oct 31 15:22:33.852983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.852901 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" event={"ID":"189fd5aa-4825-426d-bfa3-f601a4aaa71b","Type":"ContainerDied","Data":"a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc"} Oct 31 15:22:33.852983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.852967 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7" event={"ID":"189fd5aa-4825-426d-bfa3-f601a4aaa71b","Type":"ContainerDied","Data":"920b59612b8ee5d40c64fd3d12bd1f49c876fea6b70669b247f0c3ee65ea8e7a"} Oct 31 15:22:33.853091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.852996 4865 scope.go:122] "RemoveContainer" containerID="a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc" Oct 31 15:22:33.854661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.854548 4865 generic.go:356] "Generic (PLEG): container finished" podID="c8505626-7c66-4416-90c5-177d9ca407e5" containerID="632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937" exitCode=0 Oct 31 15:22:33.854661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.854591 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" Oct 31 15:22:33.854661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.854594 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" event={"ID":"c8505626-7c66-4416-90c5-177d9ca407e5","Type":"ContainerDied","Data":"632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937"} Oct 31 15:22:33.854933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.854709 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm" event={"ID":"c8505626-7c66-4416-90c5-177d9ca407e5","Type":"ContainerDied","Data":"a0422b2c95f9cc5ae0ff80ffb3f5604c4cdf2365168692868a42f309e293f4e6"} Oct 31 15:22:33.867246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.867220 4865 scope.go:122] "RemoveContainer" containerID="a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc" Oct 31 15:22:33.868020 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:22:33.867995 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc\": container with ID starting with a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc not found: ID does not exist" containerID="a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc" Oct 31 15:22:33.868141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.868032 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc"} err="failed to get container status \"a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc\": rpc error: code = NotFound desc = could not find container \"a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc\": container with ID starting with a1e1bba462d61733ac3f6ce337c5858da7c783dce8cbb56e420546a494156cdc not found: ID does not exist" Oct 31 15:22:33.868141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.868055 4865 scope.go:122] "RemoveContainer" containerID="632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937" Oct 31 15:22:33.883736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.883715 4865 scope.go:122] "RemoveContainer" containerID="632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937" Oct 31 15:22:33.884073 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:22:33.884003 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937\": container with ID starting with 632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937 not found: ID does not exist" containerID="632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937" Oct 31 15:22:33.884073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.884046 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937"} err="failed to get container status \"632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937\": rpc error: code = NotFound desc = could not find container \"632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937\": container with ID starting with 632f3da95ea10998ad39505ba5eb7773d2693807dbe706abd0928b7a61ebd937 not found: ID does not exist" Oct 31 15:22:33.912681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:33.912651 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nlwm\" (UniqueName: \"kubernetes.io/projected/189fd5aa-4825-426d-bfa3-f601a4aaa71b-kube-api-access-7nlwm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:34.087931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:34.087899 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7"] Oct 31 15:22:34.090588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:34.090566 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-379d72-5b8cb75c7f-qb9d7"] Oct 31 15:22:34.202100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:34.202027 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm"] Oct 31 15:22:34.204764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:34.204741 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-379d72-7b44c97995-j25gm"] Oct 31 15:22:34.498322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:34.498292 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189fd5aa-4825-426d-bfa3-f601a4aaa71b" path="/var/lib/kubelet/pods/189fd5aa-4825-426d-bfa3-f601a4aaa71b/volumes" Oct 31 15:22:34.498778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:34.498761 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8505626-7c66-4416-90c5-177d9ca407e5" path="/var/lib/kubelet/pods/c8505626-7c66-4416-90c5-177d9ca407e5/volumes" Oct 31 15:22:37.420311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:37.420236 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:37.637535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:37.637501 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m"] Oct 31 15:22:37.746511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:37.746461 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467"] Oct 31 15:22:37.815678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:37.815649 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:22:37.815848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:37.815722 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:22:37.997182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:37.997102 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:38.136644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:38.136613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:38.898848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:38.898818 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:42.637829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.637786 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" podUID="e6e2069a-ae4b-44c1-91df-b6e8ca86238b" containerName="payload-processing" containerID="cri-o://92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a" gracePeriod=25 Oct 31 15:22:42.747581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.747524 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" podUID="3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" containerName="payload-pre-processing" containerID="cri-o://3c853e3569fa2901b5fa4ed25bf3361a140af9a85448f12031a0cea9db611b0e" gracePeriod=25 Oct 31 15:22:42.902587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.902523 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:22:42.914879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.914853 4865 generic.go:356] "Generic (PLEG): container finished" podID="3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" containerID="3c853e3569fa2901b5fa4ed25bf3361a140af9a85448f12031a0cea9db611b0e" exitCode=0 Oct 31 15:22:42.915027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.914925 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" event={"ID":"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a","Type":"ContainerDied","Data":"3c853e3569fa2901b5fa4ed25bf3361a140af9a85448f12031a0cea9db611b0e"} Oct 31 15:22:42.916285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.916258 4865 generic.go:356] "Generic (PLEG): container finished" podID="e6e2069a-ae4b-44c1-91df-b6e8ca86238b" containerID="92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a" exitCode=0 Oct 31 15:22:42.916436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.916326 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" event={"ID":"e6e2069a-ae4b-44c1-91df-b6e8ca86238b","Type":"ContainerDied","Data":"92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a"} Oct 31 15:22:42.916436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.916372 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" event={"ID":"e6e2069a-ae4b-44c1-91df-b6e8ca86238b","Type":"ContainerDied","Data":"ce7019e63d8be4d4c7b16d5c3bc8b83ee353698ebf390d4842911f608f9c465d"} Oct 31 15:22:42.916436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.916377 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m" Oct 31 15:22:42.916436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.916393 4865 scope.go:122] "RemoveContainer" containerID="92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a" Oct 31 15:22:42.933951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.933930 4865 scope.go:122] "RemoveContainer" containerID="92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a" Oct 31 15:22:42.934309 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:22:42.934273 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a\": container with ID starting with 92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a not found: ID does not exist" containerID="92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a" Oct 31 15:22:42.934413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:42.934312 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a"} err="failed to get container status \"92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a\": rpc error: code = NotFound desc = could not find container \"92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a\": container with ID starting with 92de1b007c312e8ae8f97a93112047314a6ed7e75f2202a7e129a6854771e03a not found: ID does not exist" Oct 31 15:22:43.020698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.020673 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:22:43.023503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.023458 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-plugins-config-volume\") pod \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " Oct 31 15:22:43.023712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.023682 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-kube-api-access-m29rb\" (UniqueName: \"kubernetes.io/projected/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-kube-api-access-m29rb\") pod \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\" (UID: \"e6e2069a-ae4b-44c1-91df-b6e8ca86238b\") " Oct 31 15:22:43.024361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.024076 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-plugins-config-volume" pod "e6e2069a-ae4b-44c1-91df-b6e8ca86238b" (UID: "e6e2069a-ae4b-44c1-91df-b6e8ca86238b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:43.024361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.024275 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:43.032255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.032229 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-kube-api-access-m29rb" pod "e6e2069a-ae4b-44c1-91df-b6e8ca86238b" (UID: "e6e2069a-ae4b-44c1-91df-b6e8ca86238b"). InnerVolumeSpecName "kube-api-access-m29rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:43.124983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.124949 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-plugins-config-volume\") pod \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " Oct 31 15:22:43.125240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.125224 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-kube-api-access-5s74j\" (UniqueName: \"kubernetes.io/projected/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-kube-api-access-5s74j\") pod \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\" (UID: \"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a\") " Oct 31 15:22:43.125339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.125316 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-plugins-config-volume" pod "3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" (UID: "3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:43.125623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.125605 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29rb\" (UniqueName: \"kubernetes.io/projected/e6e2069a-ae4b-44c1-91df-b6e8ca86238b-kube-api-access-m29rb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:43.125724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.125629 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:43.127558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.127529 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-kube-api-access-5s74j" pod "3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" (UID: "3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a"). InnerVolumeSpecName "kube-api-access-5s74j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:43.226356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.226329 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s74j\" (UniqueName: \"kubernetes.io/projected/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a-kube-api-access-5s74j\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:43.305140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.305110 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:43.339922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.339891 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m"] Oct 31 15:22:43.348433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.348402 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-9dd9bcccf-zqt9m"] Oct 31 15:22:43.924557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.924444 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" Oct 31 15:22:43.924557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.924481 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" event={"ID":"3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a","Type":"ContainerDied","Data":"01207a82d637f78c0c031a57e61f5874b0f29c1491e20c746773f019acf9e853"} Oct 31 15:22:43.924557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.924538 4865 scope.go:122] "RemoveContainer" containerID="3c853e3569fa2901b5fa4ed25bf3361a140af9a85448f12031a0cea9db611b0e" Oct 31 15:22:43.934368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:43.934340 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467" podUID="3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.203:50052\" within 1s: context deadline exceeded" Oct 31 15:22:43.934613 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:22:43.934596 4865 logging.go:55] [core] [Channel #4403 SubChannel #4404]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.203:50052", ServerName: "10.132.0.203:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.203:50052: operation was canceled" Oct 31 15:22:44.098931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:44.098897 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467"] Oct 31 15:22:44.101457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:44.101438 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-5b8d5597b7-bq467"] Oct 31 15:22:44.496041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:44.496009 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" path="/var/lib/kubelet/pods/3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a/volumes" Oct 31 15:22:44.496480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:44.496445 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e2069a-ae4b-44c1-91df-b6e8ca86238b" path="/var/lib/kubelet/pods/e6e2069a-ae4b-44c1-91df-b6e8ca86238b/volumes" Oct 31 15:22:45.087163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:45.087132 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:45.278977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:45.278947 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:45.412810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:45.412733 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:47.490658 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:22:47.490622 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:22:50.078837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:50.078801 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:51.360055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:51.360023 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:51.515455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:51.515418 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:53.048529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:53.048499 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:53.526207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:53.526175 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:54.401361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:54.401330 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:56.220683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:56.220651 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:56.757250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:56.757223 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:56.892868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:56.892837 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:57.306799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:57.306768 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:58.200342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:58.200307 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:58.701996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:58.701965 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:58.733772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:58.733745 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z"] Oct 31 15:22:58.733975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:58.733951 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" containerID="cri-o://c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0" gracePeriod=30 Oct 31 15:22:58.832583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:58.832552 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:59.547686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:59.547652 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:22:59.708462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:22:59.708430 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:23:00.048858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:00.048821 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7"] Oct 31 15:23:00.051295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:00.051275 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-609721ee-89lr7"] Oct 31 15:23:00.496891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:00.496858 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09517863-9826-4062-93a6-742cce83ab09" path="/var/lib/kubelet/pods/09517863-9826-4062-93a6-742cce83ab09/volumes" Oct 31 15:23:01.552590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:01.552554 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:03.991831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:03.991809 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:23:04.055640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.055578 4865 generic.go:356] "Generic (PLEG): container finished" podID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerID="c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0" exitCode=0 Oct 31 15:23:04.055640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.055626 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" Oct 31 15:23:04.055849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.055667 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" event={"ID":"e15e1799-085c-409b-8365-6f8ea4cdeda1","Type":"ContainerDied","Data":"c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0"} Oct 31 15:23:04.055849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.055736 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z" event={"ID":"e15e1799-085c-409b-8365-6f8ea4cdeda1","Type":"ContainerDied","Data":"e593fee532bcb32915d8a47940aa43fe7fd6fd155c9d2b6c6f91f7e7d6ddad7c"} Oct 31 15:23:04.055849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.055765 4865 scope.go:122] "RemoveContainer" containerID="c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0" Oct 31 15:23:04.065669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.065652 4865 scope.go:122] "RemoveContainer" containerID="da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524" Oct 31 15:23:04.074997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.074984 4865 scope.go:122] "RemoveContainer" containerID="c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0" Oct 31 15:23:04.075266 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:23:04.075247 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0\": container with ID starting with c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0 not found: ID does not exist" containerID="c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0" Oct 31 15:23:04.075319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.075273 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0"} err="failed to get container status \"c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0\": rpc error: code = NotFound desc = could not find container \"c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0\": container with ID starting with c0f84afcfa710430bbdc53a50bedc74bb7c261bfa327bdaffdee207ac67c52c0 not found: ID does not exist" Oct 31 15:23:04.075319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.075288 4865 scope.go:122] "RemoveContainer" containerID="da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524" Oct 31 15:23:04.075533 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:23:04.075516 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524\": container with ID starting with da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524 not found: ID does not exist" containerID="da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524" Oct 31 15:23:04.075614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.075536 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524"} err="failed to get container status \"da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524\": rpc error: code = NotFound desc = could not find container \"da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524\": container with ID starting with da5d665ffc0acfadbb0e1b7c1dbc1a992c52d7caadf8ac7dcda020f800e12524 not found: ID does not exist" Oct 31 15:23:04.121979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.121955 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-envoy\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122026 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-data\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122203 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-certs\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122220 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-data" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:04.122283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122264 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-kube-api-access-9pb2s\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-kube-api-access-9pb2s\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122300 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15e1799-085c-409b-8365-6f8ea4cdeda1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15e1799-085c-409b-8365-6f8ea4cdeda1-istiod-ca-cert\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122327 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-socket\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122368 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-credential-socket\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122449 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-token\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-token\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122514 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-certs" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:04.122570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122561 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-podinfo\") pod \"e15e1799-085c-409b-8365-6f8ea4cdeda1\" (UID: \"e15e1799-085c-409b-8365-6f8ea4cdeda1\") " Oct 31 15:23:04.122782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122669 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-socket" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:04.122782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122726 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-credential-socket" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:04.122883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.122775 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15e1799-085c-409b-8365-6f8ea4cdeda1-istiod-ca-cert" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:04.123092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.123072 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.123159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.123098 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.123159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.123113 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.123159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.123131 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15e1799-085c-409b-8365-6f8ea4cdeda1-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.123159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.123145 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.124915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.124890 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-token" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:04.125068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.125051 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-envoy" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:04.125205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.125185 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-kube-api-access-9pb2s" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "kube-api-access-9pb2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:04.125265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.125226 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-podinfo" pod "e15e1799-085c-409b-8365-6f8ea4cdeda1" (UID: "e15e1799-085c-409b-8365-6f8ea4cdeda1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:23:04.224841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.224810 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.224841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.224839 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.225064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.224849 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pb2s\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-kube-api-access-9pb2s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.225064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.224859 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15e1799-085c-409b-8365-6f8ea4cdeda1-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:04.487800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.487762 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z"] Oct 31 15:23:04.496758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:04.496730 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-379d72-openshift-default-6d7fc549d9-7q27z"] Oct 31 15:23:05.521322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:05.521287 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:05.932225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:05.932147 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:06.092419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:06.092376 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj"] Oct 31 15:23:06.200102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:06.200019 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw"] Oct 31 15:23:06.444141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:06.444107 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:06.498365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:06.498333 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" path="/var/lib/kubelet/pods/e15e1799-085c-409b-8365-6f8ea4cdeda1/volumes" Oct 31 15:23:06.577386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:06.577355 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:07.837151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:07.837122 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:09.053641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.053609 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4"] Oct 31 15:23:09.054351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054329 4865 cpu_manager.go:414] "containerMap: removing container" podUID="189fd5aa-4825-426d-bfa3-f601a4aaa71b" containerName="payload-pre-processing" Oct 31 15:23:09.054351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054351 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189fd5aa-4825-426d-bfa3-f601a4aaa71b" containerName="payload-pre-processing" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054367 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e6e2069a-ae4b-44c1-91df-b6e8ca86238b" containerName="payload-processing" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054376 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e2069a-ae4b-44c1-91df-b6e8ca86238b" containerName="payload-processing" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054402 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c8505626-7c66-4416-90c5-177d9ca407e5" containerName="payload-processing" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054411 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8505626-7c66-4416-90c5-177d9ca407e5" containerName="payload-processing" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054446 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054453 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054483 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054490 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054523 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" containerName="payload-pre-processing" Oct 31 15:23:09.054560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054531 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" containerName="payload-pre-processing" Oct 31 15:23:09.055061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054671 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" Oct 31 15:23:09.055061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054687 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15e1799-085c-409b-8365-6f8ea4cdeda1" containerName="istio-proxy" Oct 31 15:23:09.055061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054702 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecf4477-0cb1-4fa5-8eaf-a9289b0d605a" containerName="payload-pre-processing" Oct 31 15:23:09.055061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054713 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="189fd5aa-4825-426d-bfa3-f601a4aaa71b" containerName="payload-pre-processing" Oct 31 15:23:09.055061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054722 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e2069a-ae4b-44c1-91df-b6e8ca86238b" containerName="payload-processing" Oct 31 15:23:09.055061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.054735 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8505626-7c66-4416-90c5-177d9ca407e5" containerName="payload-processing" Oct 31 15:23:09.060045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.060022 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.113443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.113415 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4"] Oct 31 15:23:09.160942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.160911 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:23:09.160942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.160932 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:23:09.161165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.160970 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:23:09.164787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.164770 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:23:09.223643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.223605 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgv4w\" (UniqueName: \"kubernetes.io/projected/a0e20d64-be4a-4274-84d8-d8ee73e6a092-kube-api-access-cgv4w\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.223807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.223654 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.223871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.223835 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.325456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.325364 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgv4w\" (UniqueName: \"kubernetes.io/projected/a0e20d64-be4a-4274-84d8-d8ee73e6a092-kube-api-access-cgv4w\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.325456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.325417 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.325736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.325596 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.326191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.326170 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.327858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.327840 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.384384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.384352 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgv4w\" (UniqueName: \"kubernetes.io/projected/a0e20d64-be4a-4274-84d8-d8ee73e6a092-kube-api-access-cgv4w\") pod \"maas-api-revoke-sub-1bc465a6-vd6v4\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:09.674338 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:09.674261 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:10.057638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:10.057612 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4"] Oct 31 15:23:10.058461 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:10.058425 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e20d64_be4a_4274_84d8_d8ee73e6a092.slice/crio-cee1d83eb97bee0c2bcaaba5b1eac8d747b931483f1e58605cd442685036c5b2 WatchSource:0}: Error finding container cee1d83eb97bee0c2bcaaba5b1eac8d747b931483f1e58605cd442685036c5b2: Status 404 returned error can't find the container with id cee1d83eb97bee0c2bcaaba5b1eac8d747b931483f1e58605cd442685036c5b2 Oct 31 15:23:10.109066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:10.109033 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" event={"ID":"a0e20d64-be4a-4274-84d8-d8ee73e6a092","Type":"ContainerStarted","Data":"cee1d83eb97bee0c2bcaaba5b1eac8d747b931483f1e58605cd442685036c5b2"} Oct 31 15:23:11.070816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.070787 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:11.093623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.093588 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" podUID="93677797-db5f-4973-9171-f01fbe7d20fd" containerName="payload-processing" containerID="cri-o://0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694" gracePeriod=25 Oct 31 15:23:11.118113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.118087 4865 generic.go:356] "Generic (PLEG): container finished" podID="a0e20d64-be4a-4274-84d8-d8ee73e6a092" containerID="463aa41e18f02fcb1c9066401f26f2ce7fd6c61acdb28773cb531c12ff7e47ac" exitCode=0 Oct 31 15:23:11.118220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.118180 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" event={"ID":"a0e20d64-be4a-4274-84d8-d8ee73e6a092","Type":"ContainerDied","Data":"463aa41e18f02fcb1c9066401f26f2ce7fd6c61acdb28773cb531c12ff7e47ac"} Oct 31 15:23:11.199386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.199349 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:11.200328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.200298 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" podUID="109b3301-6d14-4a0b-8cd7-e25377ef3dbd" containerName="payload-pre-processing" containerID="cri-o://08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a" gracePeriod=25 Oct 31 15:23:11.371743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.371718 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:23:11.467436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.467401 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93677797-db5f-4973-9171-f01fbe7d20fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93677797-db5f-4973-9171-f01fbe7d20fd-plugins-config-volume\") pod \"93677797-db5f-4973-9171-f01fbe7d20fd\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " Oct 31 15:23:11.467682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.467662 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93677797-db5f-4973-9171-f01fbe7d20fd-kube-api-access-bfssb\" (UniqueName: \"kubernetes.io/projected/93677797-db5f-4973-9171-f01fbe7d20fd-kube-api-access-bfssb\") pod \"93677797-db5f-4973-9171-f01fbe7d20fd\" (UID: \"93677797-db5f-4973-9171-f01fbe7d20fd\") " Oct 31 15:23:11.467838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.467811 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93677797-db5f-4973-9171-f01fbe7d20fd-plugins-config-volume" pod "93677797-db5f-4973-9171-f01fbe7d20fd" (UID: "93677797-db5f-4973-9171-f01fbe7d20fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:11.468236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.468218 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93677797-db5f-4973-9171-f01fbe7d20fd-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:11.469930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.469903 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93677797-db5f-4973-9171-f01fbe7d20fd-kube-api-access-bfssb" pod "93677797-db5f-4973-9171-f01fbe7d20fd" (UID: "93677797-db5f-4973-9171-f01fbe7d20fd"). InnerVolumeSpecName "kube-api-access-bfssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:11.505542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.505509 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:23:11.569668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.569621 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-plugins-config-volume\") pod \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " Oct 31 15:23:11.569870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.569826 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-kube-api-access-952fn\" (UniqueName: \"kubernetes.io/projected/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-kube-api-access-952fn\") pod \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\" (UID: \"109b3301-6d14-4a0b-8cd7-e25377ef3dbd\") " Oct 31 15:23:11.570024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.569997 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-plugins-config-volume" pod "109b3301-6d14-4a0b-8cd7-e25377ef3dbd" (UID: "109b3301-6d14-4a0b-8cd7-e25377ef3dbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:11.570443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.570421 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfssb\" (UniqueName: \"kubernetes.io/projected/93677797-db5f-4973-9171-f01fbe7d20fd-kube-api-access-bfssb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:11.570648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.570566 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:11.572165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.572144 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-kube-api-access-952fn" pod "109b3301-6d14-4a0b-8cd7-e25377ef3dbd" (UID: "109b3301-6d14-4a0b-8cd7-e25377ef3dbd"). InnerVolumeSpecName "kube-api-access-952fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:11.671934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:11.671901 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952fn\" (UniqueName: \"kubernetes.io/projected/109b3301-6d14-4a0b-8cd7-e25377ef3dbd-kube-api-access-952fn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:12.135776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.135744 4865 generic.go:356] "Generic (PLEG): container finished" podID="93677797-db5f-4973-9171-f01fbe7d20fd" containerID="0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694" exitCode=0 Oct 31 15:23:12.136271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.135813 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" Oct 31 15:23:12.136271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.135831 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" event={"ID":"93677797-db5f-4973-9171-f01fbe7d20fd","Type":"ContainerDied","Data":"0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694"} Oct 31 15:23:12.136271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.135916 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj" event={"ID":"93677797-db5f-4973-9171-f01fbe7d20fd","Type":"ContainerDied","Data":"6357f88556a40dc05440ddb2ed9b262fb64cf0b8f35d518230540c0c46fd8ae4"} Oct 31 15:23:12.136271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.135948 4865 scope.go:122] "RemoveContainer" containerID="0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694" Oct 31 15:23:12.137185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.137146 4865 generic.go:356] "Generic (PLEG): container finished" podID="109b3301-6d14-4a0b-8cd7-e25377ef3dbd" containerID="08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a" exitCode=0 Oct 31 15:23:12.137253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.137189 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" Oct 31 15:23:12.137253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.137225 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" event={"ID":"109b3301-6d14-4a0b-8cd7-e25377ef3dbd","Type":"ContainerDied","Data":"08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a"} Oct 31 15:23:12.137364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.137273 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw" event={"ID":"109b3301-6d14-4a0b-8cd7-e25377ef3dbd","Type":"ContainerDied","Data":"bb81d9e40dc95c8f468a4b05ef8626d2bbb7d752cde523c09957a960762dbc98"} Oct 31 15:23:12.147192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.147177 4865 scope.go:122] "RemoveContainer" containerID="0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694" Oct 31 15:23:12.147594 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:23:12.147450 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694\": container with ID starting with 0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694 not found: ID does not exist" containerID="0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694" Oct 31 15:23:12.147594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.147539 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694"} err="failed to get container status \"0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694\": rpc error: code = NotFound desc = could not find container \"0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694\": container with ID starting with 0b284e531d37b0044b23f5f6d89c2e5faeb20ff0f3bcc0c680e97ad4144b0694 not found: ID does not exist" Oct 31 15:23:12.147594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.147563 4865 scope.go:122] "RemoveContainer" containerID="08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a" Oct 31 15:23:12.160726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.160707 4865 scope.go:122] "RemoveContainer" containerID="08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a" Oct 31 15:23:12.161041 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:23:12.160987 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a\": container with ID starting with 08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a not found: ID does not exist" containerID="08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a" Oct 31 15:23:12.161041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.161018 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a"} err="failed to get container status \"08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a\": rpc error: code = NotFound desc = could not find container \"08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a\": container with ID starting with 08bd33596ceb4aa230d185c469b487de2b168467752a22b2a2f81abe20d23e3a not found: ID does not exist" Oct 31 15:23:12.280770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.280748 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:12.304782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.304754 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw"] Oct 31 15:23:12.355589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.355564 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-15cc44-84bb67c84c-tbpkw"] Oct 31 15:23:12.379413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.379386 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-openshift-ca\") pod \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " Oct 31 15:23:12.379587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.379456 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-xks-ca\") pod \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " Oct 31 15:23:12.379587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.379510 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e20d64-be4a-4274-84d8-d8ee73e6a092-kube-api-access-cgv4w\" (UniqueName: \"kubernetes.io/projected/a0e20d64-be4a-4274-84d8-d8ee73e6a092-kube-api-access-cgv4w\") pod \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\" (UID: \"a0e20d64-be4a-4274-84d8-d8ee73e6a092\") " Oct 31 15:23:12.379843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.379818 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-openshift-ca" pod "a0e20d64-be4a-4274-84d8-d8ee73e6a092" (UID: "a0e20d64-be4a-4274-84d8-d8ee73e6a092"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:12.379994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.379966 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:12.381779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.381747 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-xks-ca" pod "a0e20d64-be4a-4274-84d8-d8ee73e6a092" (UID: "a0e20d64-be4a-4274-84d8-d8ee73e6a092"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:12.381867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.381844 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e20d64-be4a-4274-84d8-d8ee73e6a092-kube-api-access-cgv4w" pod "a0e20d64-be4a-4274-84d8-d8ee73e6a092" (UID: "a0e20d64-be4a-4274-84d8-d8ee73e6a092"). InnerVolumeSpecName "kube-api-access-cgv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:12.418481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.418439 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj"] Oct 31 15:23:12.424880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.424855 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-15cc44-6f577b549c-6v7mj"] Oct 31 15:23:12.481382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.481356 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e20d64-be4a-4274-84d8-d8ee73e6a092-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:12.481382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.481382 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgv4w\" (UniqueName: \"kubernetes.io/projected/a0e20d64-be4a-4274-84d8-d8ee73e6a092-kube-api-access-cgv4w\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:12.497281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.497250 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="109b3301-6d14-4a0b-8cd7-e25377ef3dbd" path="/var/lib/kubelet/pods/109b3301-6d14-4a0b-8cd7-e25377ef3dbd/volumes" Oct 31 15:23:12.497788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.497769 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93677797-db5f-4973-9171-f01fbe7d20fd" path="/var/lib/kubelet/pods/93677797-db5f-4973-9171-f01fbe7d20fd/volumes" Oct 31 15:23:12.572827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.572743 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:12.833865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:12.833818 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:13.012914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:13.012877 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:13.026023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:13.025992 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx"] Oct 31 15:23:13.026280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:13.026246 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" containerID="cri-o://5d7dcda2bbdfa389fee9589816ecd838251433efc1f131dd071469189127664a" gracePeriod=30 Oct 31 15:23:13.183550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:13.179608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:13.225290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:13.221146 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" Oct 31 15:23:13.225290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:13.221160 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4" event={"ID":"a0e20d64-be4a-4274-84d8-d8ee73e6a092","Type":"ContainerDied","Data":"cee1d83eb97bee0c2bcaaba5b1eac8d747b931483f1e58605cd442685036c5b2"} Oct 31 15:23:13.225290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:13.222015 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cee1d83eb97bee0c2bcaaba5b1eac8d747b931483f1e58605cd442685036c5b2" Oct 31 15:23:17.489307 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:23:17.489273 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:23:18.077476 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.077430 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:18.260432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.260402 4865 generic.go:356] "Generic (PLEG): container finished" podID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerID="5d7dcda2bbdfa389fee9589816ecd838251433efc1f131dd071469189127664a" exitCode=0 Oct 31 15:23:18.260656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.260630 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" event={"ID":"539e7c13-c6e7-49ad-a03d-617a87a5157b","Type":"ContainerDied","Data":"5d7dcda2bbdfa389fee9589816ecd838251433efc1f131dd071469189127664a"} Oct 31 15:23:18.260739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.260688 4865 scope.go:122] "RemoveContainer" containerID="e28b2fd24e44a296f5bf2ae4f7863d0f13e872ec7733327d4686a6625af67cd0" Oct 31 15:23:18.328094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.328074 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:23:18.378256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378222 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-kube-api-access-jj2sz\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-kube-api-access-jj2sz\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378262 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-token\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-token\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378340 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539e7c13-c6e7-49ad-a03d-617a87a5157b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539e7c13-c6e7-49ad-a03d-617a87a5157b-istiod-ca-cert\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378409 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-data\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378440 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-podinfo\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378487 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-envoy\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378516 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-credential-socket\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378626 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-socket\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378715 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-certs\") pod \"539e7c13-c6e7-49ad-a03d-617a87a5157b\" (UID: \"539e7c13-c6e7-49ad-a03d-617a87a5157b\") " Oct 31 15:23:18.378843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378768 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-data" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:18.378911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378851 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539e7c13-c6e7-49ad-a03d-617a87a5157b-istiod-ca-cert" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:18.379020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.378997 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-credential-socket" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:18.379020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.379009 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-socket" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:18.379365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.379207 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-certs" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:18.379365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.379270 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539e7c13-c6e7-49ad-a03d-617a87a5157b-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.379365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.379291 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.379365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.379304 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.379365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.379318 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.380910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.380883 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-podinfo" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:23:18.381039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.380950 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-envoy" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:23:18.381295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.381272 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-kube-api-access-jj2sz" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "kube-api-access-jj2sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:18.381400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.381354 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-token" pod "539e7c13-c6e7-49ad-a03d-617a87a5157b" (UID: "539e7c13-c6e7-49ad-a03d-617a87a5157b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:18.480177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.480142 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.480177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.480173 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.480177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.480185 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539e7c13-c6e7-49ad-a03d-617a87a5157b-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.480432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.480195 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2sz\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-kube-api-access-jj2sz\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:18.480432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:18.480206 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539e7c13-c6e7-49ad-a03d-617a87a5157b-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:19.268371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:19.268344 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" Oct 31 15:23:19.268894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:19.268394 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx" event={"ID":"539e7c13-c6e7-49ad-a03d-617a87a5157b","Type":"ContainerDied","Data":"0e369112ab34a6fbea610938330b66cdc8d31550a0f1ba6b713e21bcebee423a"} Oct 31 15:23:19.268894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:19.268449 4865 scope.go:122] "RemoveContainer" containerID="5d7dcda2bbdfa389fee9589816ecd838251433efc1f131dd071469189127664a" Oct 31 15:23:19.487236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:19.487199 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx"] Oct 31 15:23:19.537891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:19.537815 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-15cc44-openshift-default-5d45574665-zrhtx"] Oct 31 15:23:19.937249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:19.937177 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:20.503447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:20.503416 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" path="/var/lib/kubelet/pods/539e7c13-c6e7-49ad-a03d-617a87a5157b/volumes" Oct 31 15:23:23.155505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:23.155452 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:25.161836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.161808 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7"] Oct 31 15:23:25.162345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162331 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a0e20d64-be4a-4274-84d8-d8ee73e6a092" containerName="revoke-keys" Oct 31 15:23:25.162390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162346 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e20d64-be4a-4274-84d8-d8ee73e6a092" containerName="revoke-keys" Oct 31 15:23:25.162390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162371 4865 cpu_manager.go:414] "containerMap: removing container" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" Oct 31 15:23:25.162390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162376 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" Oct 31 15:23:25.162513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162393 4865 cpu_manager.go:414] "containerMap: removing container" podUID="109b3301-6d14-4a0b-8cd7-e25377ef3dbd" containerName="payload-pre-processing" Oct 31 15:23:25.162513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162398 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="109b3301-6d14-4a0b-8cd7-e25377ef3dbd" containerName="payload-pre-processing" Oct 31 15:23:25.162513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162407 4865 cpu_manager.go:414] "containerMap: removing container" podUID="93677797-db5f-4973-9171-f01fbe7d20fd" containerName="payload-processing" Oct 31 15:23:25.162513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162413 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93677797-db5f-4973-9171-f01fbe7d20fd" containerName="payload-processing" Oct 31 15:23:25.162513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162423 4865 cpu_manager.go:414] "containerMap: removing container" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" Oct 31 15:23:25.162513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162428 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" Oct 31 15:23:25.162701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162530 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" Oct 31 15:23:25.162701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162541 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="109b3301-6d14-4a0b-8cd7-e25377ef3dbd" containerName="payload-pre-processing" Oct 31 15:23:25.162701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162547 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="93677797-db5f-4973-9171-f01fbe7d20fd" containerName="payload-processing" Oct 31 15:23:25.162701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.162555 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e20d64-be4a-4274-84d8-d8ee73e6a092" containerName="revoke-keys" Oct 31 15:23:25.167796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.167776 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.239855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.239831 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7"] Oct 31 15:23:25.268753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.268723 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.268889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.268871 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.269052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.269027 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dm2\" (UniqueName: \"kubernetes.io/projected/38f001d4-d0d3-47d0-971a-b03b3157f1fd-kube-api-access-z8dm2\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.283332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.283308 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:23:25.283332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.283308 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:23:25.283506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.283336 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:23:25.283506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.283373 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:23:25.370035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.370001 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dm2\" (UniqueName: \"kubernetes.io/projected/38f001d4-d0d3-47d0-971a-b03b3157f1fd-kube-api-access-z8dm2\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.370199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.370101 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.370280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.370203 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.370982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.370914 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.372651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.372629 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.429073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.429010 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dm2\" (UniqueName: \"kubernetes.io/projected/38f001d4-d0d3-47d0-971a-b03b3157f1fd-kube-api-access-z8dm2\") pod \"maas-api-revoke-sub-d3e1d077-pjpk7\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.481293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.481267 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:25.856382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:25.856341 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7"] Oct 31 15:23:25.857017 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:25.856992 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f001d4_d0d3_47d0_971a_b03b3157f1fd.slice/crio-2e58d66cf4d856496911554825bf68fd920739015231677807001ffc8e1f45ed WatchSource:0}: Error finding container 2e58d66cf4d856496911554825bf68fd920739015231677807001ffc8e1f45ed: Status 404 returned error can't find the container with id 2e58d66cf4d856496911554825bf68fd920739015231677807001ffc8e1f45ed Oct 31 15:23:26.312689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:26.312656 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" event={"ID":"38f001d4-d0d3-47d0-971a-b03b3157f1fd","Type":"ContainerStarted","Data":"196a19ca79ba034b2a56e5171b12b4cfeca166522610e0e8a67acadcf522e33d"} Oct 31 15:23:26.313079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:26.312706 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" event={"ID":"38f001d4-d0d3-47d0-971a-b03b3157f1fd","Type":"ContainerStarted","Data":"2e58d66cf4d856496911554825bf68fd920739015231677807001ffc8e1f45ed"} Oct 31 15:23:26.431120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:26.431030 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" podStartSLOduration=1.431017047 podStartE2EDuration="1.431017047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:26.429954722 +0000 UTC m=+2252.049506134" watchObservedRunningTime="2026-10-31 15:23:26.431017047 +0000 UTC m=+2252.050568455" Oct 31 15:23:26.690451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:26.690373 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:27.170207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.170162 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:27.321063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.321031 4865 generic.go:356] "Generic (PLEG): container finished" podID="38f001d4-d0d3-47d0-971a-b03b3157f1fd" containerID="196a19ca79ba034b2a56e5171b12b4cfeca166522610e0e8a67acadcf522e33d" exitCode=0 Oct 31 15:23:27.321623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.321109 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" event={"ID":"38f001d4-d0d3-47d0-971a-b03b3157f1fd","Type":"ContainerDied","Data":"196a19ca79ba034b2a56e5171b12b4cfeca166522610e0e8a67acadcf522e33d"} Oct 31 15:23:27.743767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.743730 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr"] Oct 31 15:23:27.744369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.744355 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="539e7c13-c6e7-49ad-a03d-617a87a5157b" containerName="istio-proxy" Oct 31 15:23:27.747812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.747783 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.812288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.812257 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5kq\" (UniqueName: \"kubernetes.io/projected/f2238581-8e2d-4216-aecb-b2d9baa08799-kube-api-access-bf5kq\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.812480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.812323 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.812480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.812366 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.844675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.844646 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr"] Oct 31 15:23:27.913809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.913772 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.913809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.913813 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.914095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.913951 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5kq\" (UniqueName: \"kubernetes.io/projected/f2238581-8e2d-4216-aecb-b2d9baa08799-kube-api-access-bf5kq\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.914520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.914462 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.916331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.916313 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:27.972462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:27.972430 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5kq\" (UniqueName: \"kubernetes.io/projected/f2238581-8e2d-4216-aecb-b2d9baa08799-kube-api-access-bf5kq\") pod \"maas-api-revoke-sub-05036612-8vghr\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:28.062633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.062572 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:28.278747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.278714 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp"] Oct 31 15:23:28.282925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.282908 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.336936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.336803 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-da9920-dockercfg-kspx5\"" Oct 31 15:23:28.340598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.340574 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp"] Oct 31 15:23:28.387525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.387497 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-da9920\"" Oct 31 15:23:28.420094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.420065 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjgrp\" (UniqueName: \"kubernetes.io/projected/f513715b-2cdc-4c65-a0a9-410f44973faa-kube-api-access-mjgrp\") pod \"payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.420262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.420235 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f513715b-2cdc-4c65-a0a9-410f44973faa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.453975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.453941 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr"] Oct 31 15:23:28.454871 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:28.454842 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2238581_8e2d_4216_aecb_b2d9baa08799.slice/crio-91ce08a7610596440027a96d69f789de224f0eaff70b0305d29d98299de0d224 WatchSource:0}: Error finding container 91ce08a7610596440027a96d69f789de224f0eaff70b0305d29d98299de0d224: Status 404 returned error can't find the container with id 91ce08a7610596440027a96d69f789de224f0eaff70b0305d29d98299de0d224 Oct 31 15:23:28.480765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.480741 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:28.521843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.521817 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f513715b-2cdc-4c65-a0a9-410f44973faa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.521984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.521936 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgrp\" (UniqueName: \"kubernetes.io/projected/f513715b-2cdc-4c65-a0a9-410f44973faa-kube-api-access-mjgrp\") pod \"payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.522575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.522559 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f513715b-2cdc-4c65-a0a9-410f44973faa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.580958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.580893 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjgrp\" (UniqueName: \"kubernetes.io/projected/f513715b-2cdc-4c65-a0a9-410f44973faa-kube-api-access-mjgrp\") pod \"payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.596553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.596490 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:28.623506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.623452 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-openshift-ca\") pod \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " Oct 31 15:23:28.623650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.623554 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f001d4-d0d3-47d0-971a-b03b3157f1fd-kube-api-access-z8dm2\" (UniqueName: \"kubernetes.io/projected/38f001d4-d0d3-47d0-971a-b03b3157f1fd-kube-api-access-z8dm2\") pod \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " Oct 31 15:23:28.623754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.623734 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-xks-ca\") pod \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\" (UID: \"38f001d4-d0d3-47d0-971a-b03b3157f1fd\") " Oct 31 15:23:28.625634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.624391 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-openshift-ca" pod "38f001d4-d0d3-47d0-971a-b03b3157f1fd" (UID: "38f001d4-d0d3-47d0-971a-b03b3157f1fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:28.627270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.627245 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:28.642357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.642320 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f001d4-d0d3-47d0-971a-b03b3157f1fd-kube-api-access-z8dm2" pod "38f001d4-d0d3-47d0-971a-b03b3157f1fd" (UID: "38f001d4-d0d3-47d0-971a-b03b3157f1fd"). InnerVolumeSpecName "kube-api-access-z8dm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:28.643983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.643945 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-xks-ca" pod "38f001d4-d0d3-47d0-971a-b03b3157f1fd" (UID: "38f001d4-d0d3-47d0-971a-b03b3157f1fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:28.742654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.729862 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dm2\" (UniqueName: \"kubernetes.io/projected/38f001d4-d0d3-47d0-971a-b03b3157f1fd-kube-api-access-z8dm2\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:28.742977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.742950 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38f001d4-d0d3-47d0-971a-b03b3157f1fd-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:28.982761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:28.982700 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp"] Oct 31 15:23:28.982899 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:28.982768 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf513715b_2cdc_4c65_a0a9_410f44973faa.slice/crio-4ebffcf337306549ded23d21e3989bb07a27205edbf77bccf848d51a6c643ec0 WatchSource:0}: Error finding container 4ebffcf337306549ded23d21e3989bb07a27205edbf77bccf848d51a6c643ec0: Status 404 returned error can't find the container with id 4ebffcf337306549ded23d21e3989bb07a27205edbf77bccf848d51a6c643ec0 Oct 31 15:23:29.045702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.045676 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:23:29.083535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.083509 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz"] Oct 31 15:23:29.084044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.084031 4865 cpu_manager.go:414] "containerMap: removing container" podUID="38f001d4-d0d3-47d0-971a-b03b3157f1fd" containerName="revoke-keys" Oct 31 15:23:29.084094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.084045 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f001d4-d0d3-47d0-971a-b03b3157f1fd" containerName="revoke-keys" Oct 31 15:23:29.084154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.084145 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f001d4-d0d3-47d0-971a-b03b3157f1fd" containerName="revoke-keys" Oct 31 15:23:29.087416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.087400 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.146148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.146120 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz"] Oct 31 15:23:29.248381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.248346 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.248566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.248406 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7qn\" (UniqueName: \"kubernetes.io/projected/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-kube-api-access-mb7qn\") pod \"payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.334909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.334875 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" event={"ID":"f513715b-2cdc-4c65-a0a9-410f44973faa","Type":"ContainerStarted","Data":"bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b"} Oct 31 15:23:29.335062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.334935 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" event={"ID":"f513715b-2cdc-4c65-a0a9-410f44973faa","Type":"ContainerStarted","Data":"4ebffcf337306549ded23d21e3989bb07a27205edbf77bccf848d51a6c643ec0"} Oct 31 15:23:29.335062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.334963 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:29.336346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.336327 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" Oct 31 15:23:29.336507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.336369 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7" event={"ID":"38f001d4-d0d3-47d0-971a-b03b3157f1fd","Type":"ContainerDied","Data":"2e58d66cf4d856496911554825bf68fd920739015231677807001ffc8e1f45ed"} Oct 31 15:23:29.336507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.336422 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e58d66cf4d856496911554825bf68fd920739015231677807001ffc8e1f45ed" Oct 31 15:23:29.338104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.338083 4865 generic.go:356] "Generic (PLEG): container finished" podID="f2238581-8e2d-4216-aecb-b2d9baa08799" containerID="e9560d4163646968f844b4e318af791191d207b708035a51c95697f355c0b23f" exitCode=0 Oct 31 15:23:29.338413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.338141 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" event={"ID":"f2238581-8e2d-4216-aecb-b2d9baa08799","Type":"ContainerDied","Data":"e9560d4163646968f844b4e318af791191d207b708035a51c95697f355c0b23f"} Oct 31 15:23:29.338413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.338174 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" event={"ID":"f2238581-8e2d-4216-aecb-b2d9baa08799","Type":"ContainerStarted","Data":"91ce08a7610596440027a96d69f789de224f0eaff70b0305d29d98299de0d224"} Oct 31 15:23:29.350009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.349985 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7qn\" (UniqueName: \"kubernetes.io/projected/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-kube-api-access-mb7qn\") pod \"payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.350213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.350194 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.350846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.350830 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.411984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.411954 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7qn\" (UniqueName: \"kubernetes.io/projected/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-kube-api-access-mb7qn\") pod \"payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.454684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.454627 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" podStartSLOduration=1.454613959 podStartE2EDuration="1.454613959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:29.451698862 +0000 UTC m=+2255.071250313" watchObservedRunningTime="2026-10-31 15:23:29.454613959 +0000 UTC m=+2255.074165364" Oct 31 15:23:29.702234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.702137 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:29.793044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:29.793005 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:30.239855 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:30.239806 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55409b0d_3c7c_49af_aeb9_fe5ffab8020a.slice/crio-9bded88468fb9cfd045215507249f325dd4be2ea42e424b04995aa72a4a74b93 WatchSource:0}: Error finding container 9bded88468fb9cfd045215507249f325dd4be2ea42e424b04995aa72a4a74b93: Status 404 returned error can't find the container with id 9bded88468fb9cfd045215507249f325dd4be2ea42e424b04995aa72a4a74b93 Oct 31 15:23:30.240781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.240756 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz"] Oct 31 15:23:30.348108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.348076 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" event={"ID":"55409b0d-3c7c-49af-aeb9-fe5ffab8020a","Type":"ContainerStarted","Data":"3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161"} Oct 31 15:23:30.348640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.348135 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" event={"ID":"55409b0d-3c7c-49af-aeb9-fe5ffab8020a","Type":"ContainerStarted","Data":"9bded88468fb9cfd045215507249f325dd4be2ea42e424b04995aa72a4a74b93"} Oct 31 15:23:30.348640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.348174 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:30.466158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.466099 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" podStartSLOduration=1.466080982 podStartE2EDuration="1.466080982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:30.465868957 +0000 UTC m=+2256.085420361" watchObservedRunningTime="2026-10-31 15:23:30.466080982 +0000 UTC m=+2256.085632394" Oct 31 15:23:30.510577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.510550 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:30.678566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.672456 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-openshift-ca\") pod \"f2238581-8e2d-4216-aecb-b2d9baa08799\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " Oct 31 15:23:30.678566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.672562 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2238581-8e2d-4216-aecb-b2d9baa08799-kube-api-access-bf5kq\" (UniqueName: \"kubernetes.io/projected/f2238581-8e2d-4216-aecb-b2d9baa08799-kube-api-access-bf5kq\") pod \"f2238581-8e2d-4216-aecb-b2d9baa08799\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " Oct 31 15:23:30.678566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.672734 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-xks-ca\") pod \"f2238581-8e2d-4216-aecb-b2d9baa08799\" (UID: \"f2238581-8e2d-4216-aecb-b2d9baa08799\") " Oct 31 15:23:30.678566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.676769 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-openshift-ca" pod "f2238581-8e2d-4216-aecb-b2d9baa08799" (UID: "f2238581-8e2d-4216-aecb-b2d9baa08799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:30.678566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.677172 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-xks-ca" pod "f2238581-8e2d-4216-aecb-b2d9baa08799" (UID: "f2238581-8e2d-4216-aecb-b2d9baa08799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:30.679919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.679890 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2238581-8e2d-4216-aecb-b2d9baa08799-kube-api-access-bf5kq" pod "f2238581-8e2d-4216-aecb-b2d9baa08799" (UID: "f2238581-8e2d-4216-aecb-b2d9baa08799"). InnerVolumeSpecName "kube-api-access-bf5kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:30.774180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.774102 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:30.774180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.774135 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2238581-8e2d-4216-aecb-b2d9baa08799-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:30.774180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:30.774147 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5kq\" (UniqueName: \"kubernetes.io/projected/f2238581-8e2d-4216-aecb-b2d9baa08799-kube-api-access-bf5kq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:31.359312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:31.359277 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" Oct 31 15:23:31.359819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:31.359323 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr" event={"ID":"f2238581-8e2d-4216-aecb-b2d9baa08799","Type":"ContainerDied","Data":"91ce08a7610596440027a96d69f789de224f0eaff70b0305d29d98299de0d224"} Oct 31 15:23:31.359819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:31.359383 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91ce08a7610596440027a96d69f789de224f0eaff70b0305d29d98299de0d224" Oct 31 15:23:34.593226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:34.593195 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:39.448760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:39.448718 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:40.350530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:40.350498 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:23:40.447480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:40.447448 4865 generic.go:356] "Generic (PLEG): container finished" podID="489b6427-e570-48d5-8ded-d0065ffc6393" containerID="82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01" exitCode=0 Oct 31 15:23:40.447657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:40.447531 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" event={"ID":"489b6427-e570-48d5-8ded-d0065ffc6393","Type":"ContainerDied","Data":"82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01"} Oct 31 15:23:40.447910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:40.447896 4865 scope.go:122] "RemoveContainer" containerID="82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01" Oct 31 15:23:40.449297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:40.449268 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:40.449613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:40.449335 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:40.449613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:40.449364 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:41.362655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:41.362626 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:23:41.455571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:41.455542 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" event={"ID":"489b6427-e570-48d5-8ded-d0065ffc6393","Type":"ContainerStarted","Data":"7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565"} Oct 31 15:23:41.757443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:41.757411 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:23:41.757665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:41.757545 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 31 15:23:41.757665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:41.757608 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 31 15:23:42.297684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.294416 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q"] Oct 31 15:23:42.319101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.319063 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f2238581-8e2d-4216-aecb-b2d9baa08799" containerName="revoke-keys" Oct 31 15:23:42.319101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.319091 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2238581-8e2d-4216-aecb-b2d9baa08799" containerName="revoke-keys" Oct 31 15:23:42.319369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.319263 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2238581-8e2d-4216-aecb-b2d9baa08799" containerName="revoke-keys" Oct 31 15:23:42.322830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.322809 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.352961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.352940 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q"] Oct 31 15:23:42.359171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.359141 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7gms\" (UniqueName: \"kubernetes.io/projected/c19a4567-d5b8-4699-a337-4470bbd810b4-kube-api-access-h7gms\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.359302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.359195 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.359556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.359534 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.376616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.376594 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:23:42.376755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.376601 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:23:42.376755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.376627 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:23:42.376880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.376643 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:23:42.460263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.460233 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.460775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.460289 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7gms\" (UniqueName: \"kubernetes.io/projected/c19a4567-d5b8-4699-a337-4470bbd810b4-kube-api-access-h7gms\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.460775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.460308 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.460961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.460938 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.462747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.462724 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.521128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.521090 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7gms\" (UniqueName: \"kubernetes.io/projected/c19a4567-d5b8-4699-a337-4470bbd810b4-kube-api-access-h7gms\") pod \"maas-api-revoke-sub-2392e8d0-whp7q\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.636461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.636372 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:42.759880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.759847 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 31 15:23:42.760067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:42.759897 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 31 15:23:43.021632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:43.021600 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q"] Oct 31 15:23:43.023404 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:43.023378 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19a4567_d5b8_4699_a337_4470bbd810b4.slice/crio-e41711f44f73beb55b38a278f3442f1a6a8ca840e062f38cc7b2b036da45314e WatchSource:0}: Error finding container e41711f44f73beb55b38a278f3442f1a6a8ca840e062f38cc7b2b036da45314e: Status 404 returned error can't find the container with id e41711f44f73beb55b38a278f3442f1a6a8ca840e062f38cc7b2b036da45314e Oct 31 15:23:43.485500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:43.484606 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" event={"ID":"c19a4567-d5b8-4699-a337-4470bbd810b4","Type":"ContainerStarted","Data":"3e801405cf71af0328a0c3868dc5d7d5529d84d4cfce3f253836065ba94a71ed"} Oct 31 15:23:43.485500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:43.484659 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" event={"ID":"c19a4567-d5b8-4699-a337-4470bbd810b4","Type":"ContainerStarted","Data":"e41711f44f73beb55b38a278f3442f1a6a8ca840e062f38cc7b2b036da45314e"} Oct 31 15:23:43.625495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:43.606886 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" podStartSLOduration=1.606866978 podStartE2EDuration="1.606866978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:43.603882864 +0000 UTC m=+2269.223434276" watchObservedRunningTime="2026-10-31 15:23:43.606866978 +0000 UTC m=+2269.226418389" Oct 31 15:23:43.758717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:43.758672 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 31 15:23:43.758942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:43.758747 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 31 15:23:44.090504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.090405 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj"] Oct 31 15:23:44.101497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.098995 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:44.114501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.109750 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.161162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.161130 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj"] Oct 31 15:23:44.200494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.181920 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.262762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.262726 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.263149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.263123 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrhh\" (UniqueName: \"kubernetes.io/projected/26ec01b3-76a8-48af-bd5f-1b6d065704f4-kube-api-access-vjrhh\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.365174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.365075 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.365174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.365136 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.365429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.365235 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrhh\" (UniqueName: \"kubernetes.io/projected/26ec01b3-76a8-48af-bd5f-1b6d065704f4-kube-api-access-vjrhh\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.365782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.365759 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.368131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.368112 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.441493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.439549 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrhh\" (UniqueName: \"kubernetes.io/projected/26ec01b3-76a8-48af-bd5f-1b6d065704f4-kube-api-access-vjrhh\") pod \"maas-api-revoke-sub-c645eee1-2jnbj\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.555440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.555397 4865 generic.go:356] "Generic (PLEG): container finished" podID="c19a4567-d5b8-4699-a337-4470bbd810b4" containerID="3e801405cf71af0328a0c3868dc5d7d5529d84d4cfce3f253836065ba94a71ed" exitCode=0 Oct 31 15:23:44.559576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.559546 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" event={"ID":"c19a4567-d5b8-4699-a337-4470bbd810b4","Type":"ContainerDied","Data":"3e801405cf71af0328a0c3868dc5d7d5529d84d4cfce3f253836065ba94a71ed"} Oct 31 15:23:44.570431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.570410 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:44.757457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.757417 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 31 15:23:44.757652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.757486 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 31 15:23:44.989286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:44.989264 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj"] Oct 31 15:23:44.991751 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:44.991723 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ec01b3_76a8_48af_bd5f_1b6d065704f4.slice/crio-a4543610c3c64f4f7bd84d0ed945704d36598b0f2b5df2dc6b63b642717ca647 WatchSource:0}: Error finding container a4543610c3c64f4f7bd84d0ed945704d36598b0f2b5df2dc6b63b642717ca647: Status 404 returned error can't find the container with id a4543610c3c64f4f7bd84d0ed945704d36598b0f2b5df2dc6b63b642717ca647 Oct 31 15:23:45.279291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:45.279263 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:45.565026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:45.564953 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" event={"ID":"26ec01b3-76a8-48af-bd5f-1b6d065704f4","Type":"ContainerStarted","Data":"2db14fdcc71d160aea5baa7e04f1adc455d55122292f90e214da1a7b892b9d16"} Oct 31 15:23:45.565026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:45.564998 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" event={"ID":"26ec01b3-76a8-48af-bd5f-1b6d065704f4","Type":"ContainerStarted","Data":"a4543610c3c64f4f7bd84d0ed945704d36598b0f2b5df2dc6b63b642717ca647"} Oct 31 15:23:45.767815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:45.758813 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 31 15:23:45.767815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:45.758873 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 31 15:23:45.767815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:45.760689 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" podStartSLOduration=1.7606709029999998 podStartE2EDuration="1.760670903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:45.734861959 +0000 UTC m=+2271.354413371" watchObservedRunningTime="2026-10-31 15:23:45.760670903 +0000 UTC m=+2271.380222315" Oct 31 15:23:45.902555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:45.885183 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv"] Oct 31 15:23:46.006585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.002017 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:46.087493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.066688 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv"] Oct 31 15:23:46.087493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.066827 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088350 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-token\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088417 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-certs\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088486 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-credential-socket\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088518 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-podinfo\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088586 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f131f3a-4de2-42b9-b875-f0d007a674fe-istiod-ca-cert\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088662 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-envoy\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088755 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-socket\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088780 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqfh\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-kube-api-access-pnqfh\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.091135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.088810 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-data\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.174536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.166922 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c4ede8-openshift-default-dockercfg-t8wrj\"" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.193911 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-credential-socket\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.193960 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-podinfo\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194045 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f131f3a-4de2-42b9-b875-f0d007a674fe-istiod-ca-cert\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194136 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-envoy\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194253 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-socket\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194282 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqfh\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-kube-api-access-pnqfh\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194311 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-data\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194352 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-token\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194396 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-certs\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.194952 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-certs\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.195594 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-credential-socket\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.199153 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-socket\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.199792 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f131f3a-4de2-42b9-b875-f0d007a674fe-istiod-ca-cert\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.204238 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-data\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.205128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.205062 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:46.254180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.253463 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:46.264953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.263089 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-podinfo\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.264953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.263446 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-envoy\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.271308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.271283 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqfh\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-kube-api-access-pnqfh\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.271647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.271627 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-token\") pod \"e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.449631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.449228 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19a4567-d5b8-4699-a337-4470bbd810b4-kube-api-access-h7gms\" (UniqueName: \"kubernetes.io/projected/c19a4567-d5b8-4699-a337-4470bbd810b4-kube-api-access-h7gms\") pod \"c19a4567-d5b8-4699-a337-4470bbd810b4\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " Oct 31 15:23:46.449631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.449446 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-xks-ca\") pod \"c19a4567-d5b8-4699-a337-4470bbd810b4\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " Oct 31 15:23:46.450531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.450038 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-openshift-ca\") pod \"c19a4567-d5b8-4699-a337-4470bbd810b4\" (UID: \"c19a4567-d5b8-4699-a337-4470bbd810b4\") " Oct 31 15:23:46.450531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.450310 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-openshift-ca" pod "c19a4567-d5b8-4699-a337-4470bbd810b4" (UID: "c19a4567-d5b8-4699-a337-4470bbd810b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:46.451024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.450779 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:46.452236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.452209 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19a4567-d5b8-4699-a337-4470bbd810b4-kube-api-access-h7gms" pod "c19a4567-d5b8-4699-a337-4470bbd810b4" (UID: "c19a4567-d5b8-4699-a337-4470bbd810b4"). InnerVolumeSpecName "kube-api-access-h7gms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:46.452703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.452679 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-xks-ca" pod "c19a4567-d5b8-4699-a337-4470bbd810b4" (UID: "c19a4567-d5b8-4699-a337-4470bbd810b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:46.552712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.552676 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7gms\" (UniqueName: \"kubernetes.io/projected/c19a4567-d5b8-4699-a337-4470bbd810b4-kube-api-access-h7gms\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:46.552712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.552711 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19a4567-d5b8-4699-a337-4470bbd810b4-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:46.571009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.570982 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:46.573126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.573101 4865 generic.go:356] "Generic (PLEG): container finished" podID="26ec01b3-76a8-48af-bd5f-1b6d065704f4" containerID="2db14fdcc71d160aea5baa7e04f1adc455d55122292f90e214da1a7b892b9d16" exitCode=0 Oct 31 15:23:46.573266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.573191 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" event={"ID":"26ec01b3-76a8-48af-bd5f-1b6d065704f4","Type":"ContainerDied","Data":"2db14fdcc71d160aea5baa7e04f1adc455d55122292f90e214da1a7b892b9d16"} Oct 31 15:23:46.574817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.574802 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" Oct 31 15:23:46.574933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.574802 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q" event={"ID":"c19a4567-d5b8-4699-a337-4470bbd810b4","Type":"ContainerDied","Data":"e41711f44f73beb55b38a278f3442f1a6a8ca840e062f38cc7b2b036da45314e"} Oct 31 15:23:46.574933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.574922 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e41711f44f73beb55b38a278f3442f1a6a8ca840e062f38cc7b2b036da45314e" Oct 31 15:23:46.877216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.877177 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:23:46.878048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.878021 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:23:46.879257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:46.879112 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:23:47.033664 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:47.033630 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f131f3a_4de2_42b9_b875_f0d007a674fe.slice/crio-b3b95557e81b8c93d888f8c6ca79e1920ba77fcf84ed5f9c4f84e60264dabc3d WatchSource:0}: Error finding container b3b95557e81b8c93d888f8c6ca79e1920ba77fcf84ed5f9c4f84e60264dabc3d: Status 404 returned error can't find the container with id b3b95557e81b8c93d888f8c6ca79e1920ba77fcf84ed5f9c4f84e60264dabc3d Oct 31 15:23:47.034515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.034488 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv"] Oct 31 15:23:47.036399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.036366 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:47.036484 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.036456 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:47.036532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.036505 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:47.491979 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:23:47.491937 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:23:47.582528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.582495 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" event={"ID":"0f131f3a-4de2-42b9-b875-f0d007a674fe","Type":"ContainerStarted","Data":"9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd"} Oct 31 15:23:47.582990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.582558 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" event={"ID":"0f131f3a-4de2-42b9-b875-f0d007a674fe","Type":"ContainerStarted","Data":"b3b95557e81b8c93d888f8c6ca79e1920ba77fcf84ed5f9c4f84e60264dabc3d"} Oct 31 15:23:47.727311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.727259 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podStartSLOduration=2.727244831 podStartE2EDuration="2.727244831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:47.725369026 +0000 UTC m=+2273.344920449" watchObservedRunningTime="2026-10-31 15:23:47.727244831 +0000 UTC m=+2273.346796249" Oct 31 15:23:47.757301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.757216 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:47.789642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.789621 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:47.877942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.877904 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-openshift-ca\") pod \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " Oct 31 15:23:47.877942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.877946 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ec01b3-76a8-48af-bd5f-1b6d065704f4-kube-api-access-vjrhh\" (UniqueName: \"kubernetes.io/projected/26ec01b3-76a8-48af-bd5f-1b6d065704f4-kube-api-access-vjrhh\") pod \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " Oct 31 15:23:47.878191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.878029 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-xks-ca\") pod \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\" (UID: \"26ec01b3-76a8-48af-bd5f-1b6d065704f4\") " Oct 31 15:23:47.878263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.878205 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-openshift-ca" pod "26ec01b3-76a8-48af-bd5f-1b6d065704f4" (UID: "26ec01b3-76a8-48af-bd5f-1b6d065704f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:47.878407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.878390 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:47.880182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.880161 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-xks-ca" pod "26ec01b3-76a8-48af-bd5f-1b6d065704f4" (UID: "26ec01b3-76a8-48af-bd5f-1b6d065704f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:47.880247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.880185 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ec01b3-76a8-48af-bd5f-1b6d065704f4-kube-api-access-vjrhh" pod "26ec01b3-76a8-48af-bd5f-1b6d065704f4" (UID: "26ec01b3-76a8-48af-bd5f-1b6d065704f4"). InnerVolumeSpecName "kube-api-access-vjrhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:47.980038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.980003 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrhh\" (UniqueName: \"kubernetes.io/projected/26ec01b3-76a8-48af-bd5f-1b6d065704f4-kube-api-access-vjrhh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:47.980038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:47.980036 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26ec01b3-76a8-48af-bd5f-1b6d065704f4-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:48.571856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.571824 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:48.577654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.577627 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:48.590521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.590493 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" Oct 31 15:23:48.590921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.590511 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj" event={"ID":"26ec01b3-76a8-48af-bd5f-1b6d065704f4","Type":"ContainerDied","Data":"a4543610c3c64f4f7bd84d0ed945704d36598b0f2b5df2dc6b63b642717ca647"} Oct 31 15:23:48.590921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.590561 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4543610c3c64f4f7bd84d0ed945704d36598b0f2b5df2dc6b63b642717ca647" Oct 31 15:23:48.590921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.590877 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:48.592137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.592118 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:23:48.917599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:48.917367 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:49.161106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:49.161070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:50.100926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:50.100884 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:50.291904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:50.291733 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:50.424691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:50.424608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:50.512630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:50.512584 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:23:50.512844 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:50.512656 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:23:50.615675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:50.615634 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:50.746187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:50.746147 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:53.059319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.059233 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx"] Oct 31 15:23:53.059815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.059801 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c19a4567-d5b8-4699-a337-4470bbd810b4" containerName="revoke-keys" Oct 31 15:23:53.059860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.059816 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19a4567-d5b8-4699-a337-4470bbd810b4" containerName="revoke-keys" Oct 31 15:23:53.059860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.059840 4865 cpu_manager.go:414] "containerMap: removing container" podUID="26ec01b3-76a8-48af-bd5f-1b6d065704f4" containerName="revoke-keys" Oct 31 15:23:53.059860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.059846 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ec01b3-76a8-48af-bd5f-1b6d065704f4" containerName="revoke-keys" Oct 31 15:23:53.059958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.059923 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19a4567-d5b8-4699-a337-4470bbd810b4" containerName="revoke-keys" Oct 31 15:23:53.059958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.059934 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ec01b3-76a8-48af-bd5f-1b6d065704f4" containerName="revoke-keys" Oct 31 15:23:53.063293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.063277 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.119083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.119056 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx"] Oct 31 15:23:53.153764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.153737 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.153946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.153774 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.153946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.153905 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7dj\" (UniqueName: \"kubernetes.io/projected/5b04c111-d276-4acb-81a3-a3ada6b44fe9-kube-api-access-8t7dj\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.165853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.165831 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:23:53.165993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.165862 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:23:53.165993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.165913 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:23:53.165993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.165918 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:23:53.255255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.255231 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.255255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.255259 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.255457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.255307 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7dj\" (UniqueName: \"kubernetes.io/projected/5b04c111-d276-4acb-81a3-a3ada6b44fe9-kube-api-access-8t7dj\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.255889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.255866 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.257875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.257850 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.313335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.313274 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7dj\" (UniqueName: \"kubernetes.io/projected/5b04c111-d276-4acb-81a3-a3ada6b44fe9-kube-api-access-8t7dj\") pod \"maas-api-revoke-sub-75d542de-8l4nx\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.377498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.377433 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:53.628254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.628163 4865 generic.go:356] "Generic (PLEG): container finished" podID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerID="173c81ec60901bb2bcd39d25a2012947e37757f69a8cf6b5396f139a4a3b4734" exitCode=0 Oct 31 15:23:53.628435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.628254 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" event={"ID":"69f26fd5-25d9-41c0-bd46-b6d03361bfe1","Type":"ContainerDied","Data":"173c81ec60901bb2bcd39d25a2012947e37757f69a8cf6b5396f139a4a3b4734"} Oct 31 15:23:53.628773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.628755 4865 scope.go:122] "RemoveContainer" containerID="173c81ec60901bb2bcd39d25a2012947e37757f69a8cf6b5396f139a4a3b4734" Oct 31 15:23:53.630538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.630498 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:53.630640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.630589 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:53.630689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.630650 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:53.653920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.653899 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:53.770820 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:53.770780 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b04c111_d276_4acb_81a3_a3ada6b44fe9.slice/crio-ffe15afbc039898c14b333927dd786b4092e650d89380f274abe7624f3767154 WatchSource:0}: Error finding container ffe15afbc039898c14b333927dd786b4092e650d89380f274abe7624f3767154: Status 404 returned error can't find the container with id ffe15afbc039898c14b333927dd786b4092e650d89380f274abe7624f3767154 Oct 31 15:23:53.833510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.824418 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx"] Oct 31 15:23:53.833510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.831994 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j"] Oct 31 15:23:53.840873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.840846 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.864144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.863674 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:53.931529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.931498 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j"] Oct 31 15:23:53.998297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998259 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998354 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5kn\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-kube-api-access-6s5kn\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998434 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998521 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998555 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998591 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998724 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998759 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:53.998854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:53.998834 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.071543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.071517 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln"] Oct 31 15:23:54.075517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.075500 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.099860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.099836 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.099989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.099875 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5kn\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-kube-api-access-6s5kn\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.099989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.099923 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.099989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.099960 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100004 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100060 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100145 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100236 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100285 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100242 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100339 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.100592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100539 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.101047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.100880 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.102764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.102740 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.102947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.102926 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.138562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.134588 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c4ede8-dockercfg-khv5t\"" Oct 31 15:23:54.163495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.160867 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c4ede8\"" Oct 31 15:23:54.195503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.189982 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln"] Oct 31 15:23:54.209287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.207748 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.209287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.209014 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h44g\" (UniqueName: \"kubernetes.io/projected/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-kube-api-access-2h44g\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.209287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.209170 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.233512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.233431 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7"] Oct 31 15:23:54.239641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.239610 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.274540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.274509 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5kn\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-kube-api-access-6s5kn\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.310640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.310608 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7"] Oct 31 15:23:54.310788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.310724 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h44g\" (UniqueName: \"kubernetes.io/projected/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-kube-api-access-2h44g\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.310893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.310876 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c7b7ef-fdb4-48da-9155-0602fcdf0482-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.310958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.310943 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.311102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.311071 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsf6d\" (UniqueName: \"kubernetes.io/projected/26c7b7ef-fdb4-48da-9155-0602fcdf0482-kube-api-access-zsf6d\") pod \"payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.311675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.311648 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.345764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.345738 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:54.370507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.370460 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h44g\" (UniqueName: \"kubernetes.io/projected/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-kube-api-access-2h44g\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.393396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.393339 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:54.419779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.419739 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsf6d\" (UniqueName: \"kubernetes.io/projected/26c7b7ef-fdb4-48da-9155-0602fcdf0482-kube-api-access-zsf6d\") pod \"payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.420388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.420289 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c7b7ef-fdb4-48da-9155-0602fcdf0482-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.438025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.437988 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c7b7ef-fdb4-48da-9155-0602fcdf0482-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.478181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.478113 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsf6d\" (UniqueName: \"kubernetes.io/projected/26c7b7ef-fdb4-48da-9155-0602fcdf0482-kube-api-access-zsf6d\") pod \"payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.528334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.528299 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:54.578747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.578710 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:54.638998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.638052 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" event={"ID":"69f26fd5-25d9-41c0-bd46-b6d03361bfe1","Type":"ContainerStarted","Data":"3071589ece45f2eac0abec6f53695e0c92a5df07414c1b97d25321b2d015fd0f"} Oct 31 15:23:54.641371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.640845 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" event={"ID":"5b04c111-d276-4acb-81a3-a3ada6b44fe9","Type":"ContainerStarted","Data":"30baee89c62657024672402bed286d673be04f567769ea5ca80295bab90371e1"} Oct 31 15:23:54.641371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.640887 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" event={"ID":"5b04c111-d276-4acb-81a3-a3ada6b44fe9","Type":"ContainerStarted","Data":"ffe15afbc039898c14b333927dd786b4092e650d89380f274abe7624f3767154"} Oct 31 15:23:54.661781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.661708 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:23:54.661943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.661914 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 15:23:54.662026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.661973 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 15:23:54.822856 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:54.822821 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd991c02_0fa9_4719_b7c6_0078ba98b9e5.slice/crio-24c92bf7c8e0d43ed036c892dda5b445a6c9e804d026ab18f9e3fadfef3db670 WatchSource:0}: Error finding container 24c92bf7c8e0d43ed036c892dda5b445a6c9e804d026ab18f9e3fadfef3db670: Status 404 returned error can't find the container with id 24c92bf7c8e0d43ed036c892dda5b445a6c9e804d026ab18f9e3fadfef3db670 Oct 31 15:23:54.824672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.824539 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln"] Oct 31 15:23:54.929083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.929033 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" podStartSLOduration=1.929018495 podStartE2EDuration="1.929018495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:54.883644928 +0000 UTC m=+2280.503196339" watchObservedRunningTime="2026-10-31 15:23:54.929018495 +0000 UTC m=+2280.548569906" Oct 31 15:23:54.929235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.929222 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j"] Oct 31 15:23:54.988675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.988653 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7"] Oct 31 15:23:54.990505 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:54.990479 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd730bc_0df8_4ecb_8c3a_2272f6aa1be3.slice/crio-723db1673175f6c53b8a2d63253cae997c050841e3cbe347fc99f21d3044728f WatchSource:0}: Error finding container 723db1673175f6c53b8a2d63253cae997c050841e3cbe347fc99f21d3044728f: Status 404 returned error can't find the container with id 723db1673175f6c53b8a2d63253cae997c050841e3cbe347fc99f21d3044728f Oct 31 15:23:54.990756 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:23:54.990739 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c7b7ef_fdb4_48da_9155_0602fcdf0482.slice/crio-a5658aba0750af995de5c0b04cb2ea84e41941de236b036a4a502e0e08d54960 WatchSource:0}: Error finding container a5658aba0750af995de5c0b04cb2ea84e41941de236b036a4a502e0e08d54960: Status 404 returned error can't find the container with id a5658aba0750af995de5c0b04cb2ea84e41941de236b036a4a502e0e08d54960 Oct 31 15:23:54.993116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.993083 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:54.993231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.993149 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:54.993231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:54.993176 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:23:55.140983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.140949 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:55.355229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.355188 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:55.511496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.510302 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:55.650349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.650265 4865 generic.go:356] "Generic (PLEG): container finished" podID="5b04c111-d276-4acb-81a3-a3ada6b44fe9" containerID="30baee89c62657024672402bed286d673be04f567769ea5ca80295bab90371e1" exitCode=0 Oct 31 15:23:55.650533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.650345 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" event={"ID":"5b04c111-d276-4acb-81a3-a3ada6b44fe9","Type":"ContainerDied","Data":"30baee89c62657024672402bed286d673be04f567769ea5ca80295bab90371e1"} Oct 31 15:23:55.651892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.651870 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" event={"ID":"26c7b7ef-fdb4-48da-9155-0602fcdf0482","Type":"ContainerStarted","Data":"a59ff0c60774026109f3cd4e3121bee92e379512eb3a809dc1565adc4bc3918c"} Oct 31 15:23:55.652018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.651909 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" event={"ID":"26c7b7ef-fdb4-48da-9155-0602fcdf0482","Type":"ContainerStarted","Data":"a5658aba0750af995de5c0b04cb2ea84e41941de236b036a4a502e0e08d54960"} Oct 31 15:23:55.652018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.651932 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:23:55.653505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.653453 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" event={"ID":"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3","Type":"ContainerStarted","Data":"e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0"} Oct 31 15:23:55.653607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.653529 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" event={"ID":"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3","Type":"ContainerStarted","Data":"723db1673175f6c53b8a2d63253cae997c050841e3cbe347fc99f21d3044728f"} Oct 31 15:23:55.655031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.654999 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" event={"ID":"fd991c02-0fa9-4719-b7c6-0078ba98b9e5","Type":"ContainerStarted","Data":"54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3"} Oct 31 15:23:55.655126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.655043 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" event={"ID":"fd991c02-0fa9-4719-b7c6-0078ba98b9e5","Type":"ContainerStarted","Data":"24c92bf7c8e0d43ed036c892dda5b445a6c9e804d026ab18f9e3fadfef3db670"} Oct 31 15:23:55.655171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.655121 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:23:55.661754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.661727 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 15:23:55.661813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.661766 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 15:23:55.923321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:55.923200 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podStartSLOduration=2.923180689 podStartE2EDuration="2.923180689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:55.922814414 +0000 UTC m=+2281.542365826" watchObservedRunningTime="2026-10-31 15:23:55.923180689 +0000 UTC m=+2281.542732102" Oct 31 15:23:56.048539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.048453 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" podStartSLOduration=2.048429376 podStartE2EDuration="2.048429376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:56.038929094 +0000 UTC m=+2281.658480506" watchObservedRunningTime="2026-10-31 15:23:56.048429376 +0000 UTC m=+2281.667980788" Oct 31 15:23:56.171137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.167613 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" podStartSLOduration=2.167590721 podStartE2EDuration="2.167590721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:56.164910591 +0000 UTC m=+2281.784462004" watchObservedRunningTime="2026-10-31 15:23:56.167590721 +0000 UTC m=+2281.787142145" Oct 31 15:23:56.534494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.533826 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:23:56.547529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.544060 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 15:23:56.547529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.544123 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 15:23:56.661343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.661308 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 15:23:56.661541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.661370 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 15:23:56.956769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.956694 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:56.960199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:56.960173 4865 scope.go:122] "RemoveContainer" containerID="e0d9f28010a7ba5982f4efd481dc2d12dbade904b4e83cce59c64626bdc29d90" Oct 31 15:23:57.077627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.077596 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-openshift-ca\") pod \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " Oct 31 15:23:57.077757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.077667 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-xks-ca\") pod \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " Oct 31 15:23:57.077851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.077839 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b04c111-d276-4acb-81a3-a3ada6b44fe9-kube-api-access-8t7dj\" (UniqueName: \"kubernetes.io/projected/5b04c111-d276-4acb-81a3-a3ada6b44fe9-kube-api-access-8t7dj\") pod \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\" (UID: \"5b04c111-d276-4acb-81a3-a3ada6b44fe9\") " Oct 31 15:23:57.078167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.077899 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-openshift-ca" pod "5b04c111-d276-4acb-81a3-a3ada6b44fe9" (UID: "5b04c111-d276-4acb-81a3-a3ada6b44fe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:57.078167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.078137 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:57.080042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.080021 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04c111-d276-4acb-81a3-a3ada6b44fe9-kube-api-access-8t7dj" pod "5b04c111-d276-4acb-81a3-a3ada6b44fe9" (UID: "5b04c111-d276-4acb-81a3-a3ada6b44fe9"). InnerVolumeSpecName "kube-api-access-8t7dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:57.080511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.080482 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-xks-ca" pod "5b04c111-d276-4acb-81a3-a3ada6b44fe9" (UID: "5b04c111-d276-4acb-81a3-a3ada6b44fe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:57.183070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.180093 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7dj\" (UniqueName: \"kubernetes.io/projected/5b04c111-d276-4acb-81a3-a3ada6b44fe9-kube-api-access-8t7dj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:57.183070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.180119 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04c111-d276-4acb-81a3-a3ada6b44fe9-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:57.533495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.529833 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 15:23:57.533495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.529892 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 15:23:57.627495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.627116 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:23:57.661855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.661818 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 15:23:57.662050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.661873 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 15:23:57.667779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.667730 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:57.669910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.669888 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" Oct 31 15:23:57.670042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.669902 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx" event={"ID":"5b04c111-d276-4acb-81a3-a3ada6b44fe9","Type":"ContainerDied","Data":"ffe15afbc039898c14b333927dd786b4092e650d89380f274abe7624f3767154"} Oct 31 15:23:57.670042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:57.669957 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe15afbc039898c14b333927dd786b4092e650d89380f274abe7624f3767154" Oct 31 15:23:58.531495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:58.530811 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 15:23:58.531495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:58.530875 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 15:23:58.668491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:58.661766 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 15:23:58.668491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:58.661829 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 15:23:59.176838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:59.176804 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:59.532496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:59.531822 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 15:23:59.532496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:59.531887 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 15:23:59.630493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:59.629700 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:23:59.661592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:59.661543 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 31 15:23:59.661776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:23:59.661603 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 31 15:24:00.530359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:00.530323 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 15:24:00.530879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:00.530848 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 15:24:00.773087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:00.771667 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:24:00.773087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:00.772357 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:24:00.773711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:00.773691 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:24:01.185314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.185278 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:01.426261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.426200 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:01.427807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.427780 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn"] Oct 31 15:24:01.428401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.428381 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5b04c111-d276-4acb-81a3-a3ada6b44fe9" containerName="revoke-keys" Oct 31 15:24:01.428499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.428402 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b04c111-d276-4acb-81a3-a3ada6b44fe9" containerName="revoke-keys" Oct 31 15:24:01.428607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.428592 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b04c111-d276-4acb-81a3-a3ada6b44fe9" containerName="revoke-keys" Oct 31 15:24:01.433743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.433720 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.489213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.489180 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:24:01.492613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.490576 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:24:01.492849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.490807 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:24:01.492985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.490889 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:24:01.498707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.498683 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn"] Oct 31 15:24:01.547495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.544832 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 15:24:01.547495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.544900 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 15:24:01.622446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.622369 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.622446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.622445 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.622691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.622659 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phc2w\" (UniqueName: \"kubernetes.io/projected/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-kube-api-access-phc2w\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.772738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.770601 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phc2w\" (UniqueName: \"kubernetes.io/projected/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-kube-api-access-phc2w\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.772738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.770699 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.772738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.770788 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.772738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.771656 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.792352 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.792316 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:01.830383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:01.830352 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phc2w\" (UniqueName: \"kubernetes.io/projected/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-kube-api-access-phc2w\") pod \"maas-api-revoke-sub-3576b9ee-j7qhn\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:02.047080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:02.047005 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:02.426924 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:02.426894 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf64a25d_11c1_4e7e_a956_bffd0ebf0a13.slice/crio-26d26a59a797c51fda8782a2bd6542f76bdfbaa6dcf92ee4199a8b4c9fd51ba5 WatchSource:0}: Error finding container 26d26a59a797c51fda8782a2bd6542f76bdfbaa6dcf92ee4199a8b4c9fd51ba5: Status 404 returned error can't find the container with id 26d26a59a797c51fda8782a2bd6542f76bdfbaa6dcf92ee4199a8b4c9fd51ba5 Oct 31 15:24:02.427163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:02.427136 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn"] Oct 31 15:24:02.530609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:02.530573 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 15:24:02.530753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:02.530636 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 15:24:02.806599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:02.806566 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" event={"ID":"df64a25d-11c1-4e7e-a956-bffd0ebf0a13","Type":"ContainerStarted","Data":"243c155c737486cc80c7ccf859777b97881adc012461708fc140f15793899b56"} Oct 31 15:24:02.806978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:02.806615 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" event={"ID":"df64a25d-11c1-4e7e-a956-bffd0ebf0a13","Type":"ContainerStarted","Data":"26d26a59a797c51fda8782a2bd6542f76bdfbaa6dcf92ee4199a8b4c9fd51ba5"} Oct 31 15:24:02.924218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:02.924165 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" podStartSLOduration=1.92415155 podStartE2EDuration="1.92415155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:02.92094307 +0000 UTC m=+2288.540494480" watchObservedRunningTime="2026-10-31 15:24:02.92415155 +0000 UTC m=+2288.543702961" Oct 31 15:24:03.814586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:03.814547 4865 generic.go:356] "Generic (PLEG): container finished" podID="df64a25d-11c1-4e7e-a956-bffd0ebf0a13" containerID="243c155c737486cc80c7ccf859777b97881adc012461708fc140f15793899b56" exitCode=0 Oct 31 15:24:03.814993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:03.814631 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" event={"ID":"df64a25d-11c1-4e7e-a956-bffd0ebf0a13","Type":"ContainerDied","Data":"243c155c737486cc80c7ccf859777b97881adc012461708fc140f15793899b56"} Oct 31 15:24:04.571639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:04.571599 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:24:04.571866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:04.571673 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:04.631812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:04.631782 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:05.004583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.004550 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:05.134233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.132606 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-openshift-ca\") pod \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " Oct 31 15:24:05.134233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.132764 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-kube-api-access-phc2w\" (UniqueName: \"kubernetes.io/projected/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-kube-api-access-phc2w\") pod \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " Oct 31 15:24:05.134233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.132819 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-xks-ca\") pod \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\" (UID: \"df64a25d-11c1-4e7e-a956-bffd0ebf0a13\") " Oct 31 15:24:05.134233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.132991 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-openshift-ca" pod "df64a25d-11c1-4e7e-a956-bffd0ebf0a13" (UID: "df64a25d-11c1-4e7e-a956-bffd0ebf0a13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:05.134233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.133360 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:05.146492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.138706 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-xks-ca" pod "df64a25d-11c1-4e7e-a956-bffd0ebf0a13" (UID: "df64a25d-11c1-4e7e-a956-bffd0ebf0a13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:24:05.146492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.139174 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-kube-api-access-phc2w" pod "df64a25d-11c1-4e7e-a956-bffd0ebf0a13" (UID: "df64a25d-11c1-4e7e-a956-bffd0ebf0a13"). InnerVolumeSpecName "kube-api-access-phc2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:05.234905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.234796 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phc2w\" (UniqueName: \"kubernetes.io/projected/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-kube-api-access-phc2w\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:05.234905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.234849 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df64a25d-11c1-4e7e-a956-bffd0ebf0a13-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:05.594971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.594936 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:24:05.595214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.595023 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:05.604336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.604304 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd"] Oct 31 15:24:05.604989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.604969 4865 cpu_manager.go:414] "containerMap: removing container" podUID="df64a25d-11c1-4e7e-a956-bffd0ebf0a13" containerName="revoke-keys" Oct 31 15:24:05.605106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.604992 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df64a25d-11c1-4e7e-a956-bffd0ebf0a13" containerName="revoke-keys" Oct 31 15:24:05.605209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.605195 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="df64a25d-11c1-4e7e-a956-bffd0ebf0a13" containerName="revoke-keys" Oct 31 15:24:05.608557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.608535 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.644484 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.644444 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:24:05.644799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.644776 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:24:05.646854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.646832 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:24:05.666966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.666912 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd"] Oct 31 15:24:05.712453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.712421 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-190693-openshift-default-dockercfg-qdhg8\"" Oct 31 15:24:05.741746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.741716 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.741942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.741759 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.741942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.741783 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.742072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.742041 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5rh\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-kube-api-access-wj5rh\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.742133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.742086 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.742194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.742155 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.742279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.742241 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.742279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.742264 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.742383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.742299 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.831070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.831045 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" Oct 31 15:24:05.831264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.831075 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn" event={"ID":"df64a25d-11c1-4e7e-a956-bffd0ebf0a13","Type":"ContainerDied","Data":"26d26a59a797c51fda8782a2bd6542f76bdfbaa6dcf92ee4199a8b4c9fd51ba5"} Oct 31 15:24:05.831264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.831127 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d26a59a797c51fda8782a2bd6542f76bdfbaa6dcf92ee4199a8b4c9fd51ba5" Oct 31 15:24:05.843630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843611 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.843750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843638 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.843750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843664 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.843750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843709 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.843750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843742 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.843923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843768 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.843970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843915 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5rh\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-kube-api-access-wj5rh\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.843970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843951 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.844085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.843982 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.844085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.844063 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.844201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.844103 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.844262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.844229 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.844321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.844279 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.844427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.844411 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.846231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.846177 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.846459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.846439 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.903095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.903058 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5rh\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-kube-api-access-wj5rh\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.903095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.903092 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-nxbcd\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:05.926300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:05.926270 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:06.361448 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:06.361403 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c295cf7_ae6a_4797_ac03_87176fa70cc6.slice/crio-be58892761caf67a5fa51c3ce587ad83a279fc2d49b2d41696d783094c32faf3 WatchSource:0}: Error finding container be58892761caf67a5fa51c3ce587ad83a279fc2d49b2d41696d783094c32faf3: Status 404 returned error can't find the container with id be58892761caf67a5fa51c3ce587ad83a279fc2d49b2d41696d783094c32faf3 Oct 31 15:24:06.362251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.362220 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd"] Oct 31 15:24:06.364096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.364058 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:24:06.364208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.364138 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:24:06.364208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.364184 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:24:06.665655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.665575 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:24:06.665824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.665788 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:24:06.843148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.843110 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" event={"ID":"5c295cf7-ae6a-4797-ac03-87176fa70cc6","Type":"ContainerStarted","Data":"964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798"} Oct 31 15:24:06.843366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.843345 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" event={"ID":"5c295cf7-ae6a-4797-ac03-87176fa70cc6","Type":"ContainerStarted","Data":"be58892761caf67a5fa51c3ce587ad83a279fc2d49b2d41696d783094c32faf3"} Oct 31 15:24:06.933614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:06.933524 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:07.027381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:07.027301 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podStartSLOduration=2.02727643 podStartE2EDuration="2.02727643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:07.021577008 +0000 UTC m=+2292.641128421" watchObservedRunningTime="2026-10-31 15:24:07.02727643 +0000 UTC m=+2292.646828024" Oct 31 15:24:07.929151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:07.929117 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 15:24:07.929620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:07.929169 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 15:24:08.927922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.927886 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 15:24:08.928154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.927952 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 15:24:08.929364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.929335 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595"] Oct 31 15:24:08.933842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.933814 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.985603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.985573 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a6ff32-2bcb-4688-9844-48856a844dc7-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.985603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.985608 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.985919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.985887 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.986120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.986100 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88b46\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-kube-api-access-88b46\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.986198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.986140 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.986399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.986380 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.986500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.986415 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.986500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.986437 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:08.986622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:08.986537 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087540 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88b46\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-kube-api-access-88b46\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087580 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087649 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087671 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087694 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087731 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087809 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a6ff32-2bcb-4688-9844-48856a844dc7-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.087834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087834 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.088261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.087900 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.088261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.088103 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.088261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.088168 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.088437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.088371 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.088681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.088452 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.088681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.088589 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a6ff32-2bcb-4688-9844-48856a844dc7-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.090225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.090191 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.090513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.090492 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.135700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.135668 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595"] Oct 31 15:24:09.146309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.146280 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.146502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.146443 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88b46\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-kube-api-access-88b46\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.251534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.251498 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:09.332900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.329971 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4"] Oct 31 15:24:09.336032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.335870 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.390256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.389990 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:24:09.390256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.390126 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:24:09.390256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.390162 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:24:09.390256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.390211 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:24:09.393460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.393041 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4"] Oct 31 15:24:09.393460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.393375 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.399729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.393558 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.399729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.393799 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpgzp\" (UniqueName: \"kubernetes.io/projected/4a43aa0f-3730-4d71-b414-e44ae053f8ee-kube-api-access-rpgzp\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.437692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.437588 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:09.527544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.514316 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.527544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.514397 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.527544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.514596 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpgzp\" (UniqueName: \"kubernetes.io/projected/4a43aa0f-3730-4d71-b414-e44ae053f8ee-kube-api-access-rpgzp\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.527544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.515683 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.527544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.525865 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.572735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.572708 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpgzp\" (UniqueName: \"kubernetes.io/projected/4a43aa0f-3730-4d71-b414-e44ae053f8ee-kube-api-access-rpgzp\") pod \"maas-api-revoke-sub-d6ba2273-z8rz4\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.665433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.665398 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:09.725505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.724426 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595"] Oct 31 15:24:09.725505 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:09.724909 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a6ff32_2bcb_4688_9844_48856a844dc7.slice/crio-26afad7a1fd5b4c21dad5d05632428f325bdc251984dcbbf437b2c9b2f048bdd WatchSource:0}: Error finding container 26afad7a1fd5b4c21dad5d05632428f325bdc251984dcbbf437b2c9b2f048bdd: Status 404 returned error can't find the container with id 26afad7a1fd5b4c21dad5d05632428f325bdc251984dcbbf437b2c9b2f048bdd Oct 31 15:24:09.727681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.727643 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:24:09.727772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.727713 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:24:09.727772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.727743 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:24:09.869227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.868584 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" event={"ID":"a7a6ff32-2bcb-4688-9844-48856a844dc7","Type":"ContainerStarted","Data":"e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15"} Oct 31 15:24:09.869227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.868646 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" event={"ID":"a7a6ff32-2bcb-4688-9844-48856a844dc7","Type":"ContainerStarted","Data":"26afad7a1fd5b4c21dad5d05632428f325bdc251984dcbbf437b2c9b2f048bdd"} Oct 31 15:24:09.927952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.927867 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 15:24:09.927952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.927932 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 15:24:09.989830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:09.989733 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podStartSLOduration=1.9897168669999998 podStartE2EDuration="1.989716867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:09.986198191 +0000 UTC m=+2295.605749603" watchObservedRunningTime="2026-10-31 15:24:09.989716867 +0000 UTC m=+2295.609268277" Oct 31 15:24:10.077365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.077332 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4"] Oct 31 15:24:10.078762 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:10.078737 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a43aa0f_3730_4d71_b414_e44ae053f8ee.slice/crio-78769f15a417331eb64c9695aac2a4d9b6407886ab75fbcb719eddfdd76b177a WatchSource:0}: Error finding container 78769f15a417331eb64c9695aac2a4d9b6407886ab75fbcb719eddfdd76b177a: Status 404 returned error can't find the container with id 78769f15a417331eb64c9695aac2a4d9b6407886ab75fbcb719eddfdd76b177a Oct 31 15:24:10.252660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.252617 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:10.253855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.253833 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 15:24:10.253980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.253884 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 15:24:10.878187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.878145 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" event={"ID":"4a43aa0f-3730-4d71-b414-e44ae053f8ee","Type":"ContainerStarted","Data":"21f6c9e4bda8997581cd2bd0b583a9af24766fb4df14fb0568cf6370b71142b8"} Oct 31 15:24:10.878427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.878207 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" event={"ID":"4a43aa0f-3730-4d71-b414-e44ae053f8ee","Type":"ContainerStarted","Data":"78769f15a417331eb64c9695aac2a4d9b6407886ab75fbcb719eddfdd76b177a"} Oct 31 15:24:10.934494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.933907 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 15:24:10.934494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:10.933976 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 15:24:11.002495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:11.000702 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" podStartSLOduration=2.000681703 podStartE2EDuration="2.000681703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:10.997937345 +0000 UTC m=+2296.617488758" watchObservedRunningTime="2026-10-31 15:24:11.000681703 +0000 UTC m=+2296.620233116" Oct 31 15:24:11.252000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:11.251962 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 15:24:11.252201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:11.252016 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 15:24:11.886424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:11.886390 4865 generic.go:356] "Generic (PLEG): container finished" podID="4a43aa0f-3730-4d71-b414-e44ae053f8ee" containerID="21f6c9e4bda8997581cd2bd0b583a9af24766fb4df14fb0568cf6370b71142b8" exitCode=0 Oct 31 15:24:11.886693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:11.886668 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" event={"ID":"4a43aa0f-3730-4d71-b414-e44ae053f8ee","Type":"ContainerDied","Data":"21f6c9e4bda8997581cd2bd0b583a9af24766fb4df14fb0568cf6370b71142b8"} Oct 31 15:24:11.936494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:11.935950 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 15:24:11.936494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:11.936025 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 15:24:12.252402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:12.252363 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 15:24:12.252828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:12.252415 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 15:24:12.927147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:12.927118 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 15:24:12.927340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:12.927168 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 15:24:13.127010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.126970 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:13.179234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.179158 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-openshift-ca\") pod \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " Oct 31 15:24:13.179234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.179225 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a43aa0f-3730-4d71-b414-e44ae053f8ee-kube-api-access-rpgzp\" (UniqueName: \"kubernetes.io/projected/4a43aa0f-3730-4d71-b414-e44ae053f8ee-kube-api-access-rpgzp\") pod \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " Oct 31 15:24:13.179457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.179351 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-xks-ca\") pod \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\" (UID: \"4a43aa0f-3730-4d71-b414-e44ae053f8ee\") " Oct 31 15:24:13.179634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.179606 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-openshift-ca" pod "4a43aa0f-3730-4d71-b414-e44ae053f8ee" (UID: "4a43aa0f-3730-4d71-b414-e44ae053f8ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:13.179880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.179860 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:13.181629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.181602 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-xks-ca" pod "4a43aa0f-3730-4d71-b414-e44ae053f8ee" (UID: "4a43aa0f-3730-4d71-b414-e44ae053f8ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:24:13.181799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.181762 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a43aa0f-3730-4d71-b414-e44ae053f8ee-kube-api-access-rpgzp" pod "4a43aa0f-3730-4d71-b414-e44ae053f8ee" (UID: "4a43aa0f-3730-4d71-b414-e44ae053f8ee"). InnerVolumeSpecName "kube-api-access-rpgzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:13.251986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.251954 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 15:24:13.252186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.252011 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 15:24:13.281920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.281881 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpgzp\" (UniqueName: \"kubernetes.io/projected/4a43aa0f-3730-4d71-b414-e44ae053f8ee-kube-api-access-rpgzp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:13.281920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.281920 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43aa0f-3730-4d71-b414-e44ae053f8ee-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:13.524247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.524207 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:13.917545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.915978 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" event={"ID":"4a43aa0f-3730-4d71-b414-e44ae053f8ee","Type":"ContainerDied","Data":"78769f15a417331eb64c9695aac2a4d9b6407886ab75fbcb719eddfdd76b177a"} Oct 31 15:24:13.917545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.916044 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78769f15a417331eb64c9695aac2a4d9b6407886ab75fbcb719eddfdd76b177a" Oct 31 15:24:13.917545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.916115 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4" Oct 31 15:24:13.928824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.928788 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Oct 31 15:24:13.928980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:13.928844 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Oct 31 15:24:14.252714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:14.252673 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 15:24:14.252907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:14.252740 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 15:24:14.773720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:14.773694 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:14.932837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:14.932807 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:14.933550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:14.933531 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:14.934215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:14.934198 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:24:15.252708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:15.252673 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 15:24:15.253003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:15.252972 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 15:24:16.251973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:16.251933 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 15:24:16.252371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:16.252002 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 15:24:17.234189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.234158 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:17.287501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.287445 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8"] Oct 31 15:24:17.288437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.288414 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4a43aa0f-3730-4d71-b414-e44ae053f8ee" containerName="revoke-keys" Oct 31 15:24:17.288493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.288442 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a43aa0f-3730-4d71-b414-e44ae053f8ee" containerName="revoke-keys" Oct 31 15:24:17.288677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.288663 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a43aa0f-3730-4d71-b414-e44ae053f8ee" containerName="revoke-keys" Oct 31 15:24:17.300555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.297889 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.345414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.345384 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8"] Oct 31 15:24:17.366135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.366095 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.366334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.366311 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzk9\" (UniqueName: \"kubernetes.io/projected/4d7af2eb-aa55-4f08-b71f-6023f92018eb-kube-api-access-4wzk9\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.366418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.366365 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.374096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.374066 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:17.392667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.392641 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:24:17.392667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.392657 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:24:17.392859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.392831 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:24:17.403175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.403143 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:24:17.467499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.467457 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzk9\" (UniqueName: \"kubernetes.io/projected/4d7af2eb-aa55-4f08-b71f-6023f92018eb-kube-api-access-4wzk9\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.467656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.467524 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.467732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.467704 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.468426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.468399 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.470017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.469998 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.493612 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:24:17.493537 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:24:17.525992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.525962 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzk9\" (UniqueName: \"kubernetes.io/projected/4d7af2eb-aa55-4f08-b71f-6023f92018eb-kube-api-access-4wzk9\") pod \"maas-api-revoke-sub-d270a049-fpsv8\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.622498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.622445 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:17.906779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.905210 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:17.906779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.905817 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:17.907414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:17.907391 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:24:18.004325 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:18.004297 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7af2eb_aa55_4f08_b71f_6023f92018eb.slice/crio-df2003af054c7eb4237d37ffb8cf95bb234cc73a855ae025c13e51e02529c6b3 WatchSource:0}: Error finding container df2003af054c7eb4237d37ffb8cf95bb234cc73a855ae025c13e51e02529c6b3: Status 404 returned error can't find the container with id df2003af054c7eb4237d37ffb8cf95bb234cc73a855ae025c13e51e02529c6b3 Oct 31 15:24:18.015549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:18.015527 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8"] Oct 31 15:24:18.962217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:18.962179 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" event={"ID":"4d7af2eb-aa55-4f08-b71f-6023f92018eb","Type":"ContainerStarted","Data":"033b7eda6ef5153b18cdd8e28386e73ab5e66e716fbd1c5c343e78badb229cbf"} Oct 31 15:24:18.962685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:18.962247 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" event={"ID":"4d7af2eb-aa55-4f08-b71f-6023f92018eb","Type":"ContainerStarted","Data":"df2003af054c7eb4237d37ffb8cf95bb234cc73a855ae025c13e51e02529c6b3"} Oct 31 15:24:19.062716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:19.062677 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:19.077919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:19.077870 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" podStartSLOduration=2.077856446 podStartE2EDuration="2.077856446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:19.076783891 +0000 UTC m=+2304.696335302" watchObservedRunningTime="2026-10-31 15:24:19.077856446 +0000 UTC m=+2304.697407856" Oct 31 15:24:19.970618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:19.970587 4865 generic.go:356] "Generic (PLEG): container finished" podID="4d7af2eb-aa55-4f08-b71f-6023f92018eb" containerID="033b7eda6ef5153b18cdd8e28386e73ab5e66e716fbd1c5c343e78badb229cbf" exitCode=0 Oct 31 15:24:19.971056 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:19.970623 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" event={"ID":"4d7af2eb-aa55-4f08-b71f-6023f92018eb","Type":"ContainerDied","Data":"033b7eda6ef5153b18cdd8e28386e73ab5e66e716fbd1c5c343e78badb229cbf"} Oct 31 15:24:20.142250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.142217 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8"] Oct 31 15:24:20.146458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.146437 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.201239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.201217 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8"] Oct 31 15:24:20.204455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.204425 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.204607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.204554 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.204746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.204726 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7l5x\" (UniqueName: \"kubernetes.io/projected/ba4bfdd6-a901-4bdc-9214-37c101e83dde-kube-api-access-j7l5x\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.306209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.306134 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.306209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.306204 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.306693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.306666 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7l5x\" (UniqueName: \"kubernetes.io/projected/ba4bfdd6-a901-4bdc-9214-37c101e83dde-kube-api-access-j7l5x\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.306983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.306957 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.308936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.308914 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.364574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.364542 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7l5x\" (UniqueName: \"kubernetes.io/projected/ba4bfdd6-a901-4bdc-9214-37c101e83dde-kube-api-access-j7l5x\") pod \"maas-api-revoke-sub-56d26e41-6mbg8\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.460354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.460322 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:20.844999 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:20.844967 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4bfdd6_a901_4bdc_9214_37c101e83dde.slice/crio-e811300ca142a8bf7091d65b22ca1acbfe8e7b500a1339833ca34f9f4bee0b3f WatchSource:0}: Error finding container e811300ca142a8bf7091d65b22ca1acbfe8e7b500a1339833ca34f9f4bee0b3f: Status 404 returned error can't find the container with id e811300ca142a8bf7091d65b22ca1acbfe8e7b500a1339833ca34f9f4bee0b3f Oct 31 15:24:20.846587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.846561 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8"] Oct 31 15:24:20.978615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:20.978579 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" event={"ID":"ba4bfdd6-a901-4bdc-9214-37c101e83dde","Type":"ContainerStarted","Data":"e811300ca142a8bf7091d65b22ca1acbfe8e7b500a1339833ca34f9f4bee0b3f"} Oct 31 15:24:21.137771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.137747 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:21.254985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.254951 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-openshift-ca\") pod \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " Oct 31 15:24:21.255204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.255048 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-xks-ca\") pod \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " Oct 31 15:24:21.255362 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.255342 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7af2eb-aa55-4f08-b71f-6023f92018eb-kube-api-access-4wzk9\" (UniqueName: \"kubernetes.io/projected/4d7af2eb-aa55-4f08-b71f-6023f92018eb-kube-api-access-4wzk9\") pod \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\" (UID: \"4d7af2eb-aa55-4f08-b71f-6023f92018eb\") " Oct 31 15:24:21.255519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.255440 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-openshift-ca" pod "4d7af2eb-aa55-4f08-b71f-6023f92018eb" (UID: "4d7af2eb-aa55-4f08-b71f-6023f92018eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:21.255917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.255890 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:21.270223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.270172 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:21.293922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.293888 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-xks-ca" pod "4d7af2eb-aa55-4f08-b71f-6023f92018eb" (UID: "4d7af2eb-aa55-4f08-b71f-6023f92018eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:24:21.294055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.293981 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7af2eb-aa55-4f08-b71f-6023f92018eb-kube-api-access-4wzk9" pod "4d7af2eb-aa55-4f08-b71f-6023f92018eb" (UID: "4d7af2eb-aa55-4f08-b71f-6023f92018eb"). InnerVolumeSpecName "kube-api-access-4wzk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:21.357524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.357496 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzk9\" (UniqueName: \"kubernetes.io/projected/4d7af2eb-aa55-4f08-b71f-6023f92018eb-kube-api-access-4wzk9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:21.357626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.357530 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7af2eb-aa55-4f08-b71f-6023f92018eb-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:21.987243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.987211 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" event={"ID":"4d7af2eb-aa55-4f08-b71f-6023f92018eb","Type":"ContainerDied","Data":"df2003af054c7eb4237d37ffb8cf95bb234cc73a855ae025c13e51e02529c6b3"} Oct 31 15:24:21.987700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.987263 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df2003af054c7eb4237d37ffb8cf95bb234cc73a855ae025c13e51e02529c6b3" Oct 31 15:24:21.987700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.987226 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8" Oct 31 15:24:21.989075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.989050 4865 generic.go:356] "Generic (PLEG): container finished" podID="ba4bfdd6-a901-4bdc-9214-37c101e83dde" containerID="79360ba0a83df46bba777778693e69a01effaa17ee35cb6a413dfb21a106a1b6" exitCode=0 Oct 31 15:24:21.989161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:21.989137 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" event={"ID":"ba4bfdd6-a901-4bdc-9214-37c101e83dde","Type":"ContainerDied","Data":"79360ba0a83df46bba777778693e69a01effaa17ee35cb6a413dfb21a106a1b6"} Oct 31 15:24:23.159792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.159764 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:23.278812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.278732 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-xks-ca\") pod \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " Oct 31 15:24:23.279003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.278867 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4bfdd6-a901-4bdc-9214-37c101e83dde-kube-api-access-j7l5x\" (UniqueName: \"kubernetes.io/projected/ba4bfdd6-a901-4bdc-9214-37c101e83dde-kube-api-access-j7l5x\") pod \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " Oct 31 15:24:23.279003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.278966 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-openshift-ca\") pod \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\" (UID: \"ba4bfdd6-a901-4bdc-9214-37c101e83dde\") " Oct 31 15:24:23.279654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.279627 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-openshift-ca" pod "ba4bfdd6-a901-4bdc-9214-37c101e83dde" (UID: "ba4bfdd6-a901-4bdc-9214-37c101e83dde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:23.281825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.281795 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4bfdd6-a901-4bdc-9214-37c101e83dde-kube-api-access-j7l5x" pod "ba4bfdd6-a901-4bdc-9214-37c101e83dde" (UID: "ba4bfdd6-a901-4bdc-9214-37c101e83dde"). InnerVolumeSpecName "kube-api-access-j7l5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:23.284875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.284848 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-xks-ca" pod "ba4bfdd6-a901-4bdc-9214-37c101e83dde" (UID: "ba4bfdd6-a901-4bdc-9214-37c101e83dde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:24:23.380948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.380918 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:23.380948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.380944 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7l5x\" (UniqueName: \"kubernetes.io/projected/ba4bfdd6-a901-4bdc-9214-37c101e83dde-kube-api-access-j7l5x\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:23.381146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:23.380956 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4bfdd6-a901-4bdc-9214-37c101e83dde-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:24.003987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:24.003948 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" event={"ID":"ba4bfdd6-a901-4bdc-9214-37c101e83dde","Type":"ContainerDied","Data":"e811300ca142a8bf7091d65b22ca1acbfe8e7b500a1339833ca34f9f4bee0b3f"} Oct 31 15:24:24.003987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:24.003989 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8" Oct 31 15:24:24.004200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:24.004006 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e811300ca142a8bf7091d65b22ca1acbfe8e7b500a1339833ca34f9f4bee0b3f" Oct 31 15:24:25.766868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:25.766837 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:30.803949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:30.803914 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:35.394071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:35.394036 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:37.775938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:37.775906 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:37.894701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:37.894664 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:38.324269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.324234 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb"] Oct 31 15:24:38.324841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.324827 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4d7af2eb-aa55-4f08-b71f-6023f92018eb" containerName="revoke-keys" Oct 31 15:24:38.324900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.324842 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7af2eb-aa55-4f08-b71f-6023f92018eb" containerName="revoke-keys" Oct 31 15:24:38.324900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.324860 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ba4bfdd6-a901-4bdc-9214-37c101e83dde" containerName="revoke-keys" Oct 31 15:24:38.324900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.324865 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4bfdd6-a901-4bdc-9214-37c101e83dde" containerName="revoke-keys" Oct 31 15:24:38.325011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.325001 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7af2eb-aa55-4f08-b71f-6023f92018eb" containerName="revoke-keys" Oct 31 15:24:38.325048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.325014 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4bfdd6-a901-4bdc-9214-37c101e83dde" containerName="revoke-keys" Oct 31 15:24:38.329709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.329691 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.384269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.384242 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb"] Oct 31 15:24:38.430746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.430710 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:24:38.430926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.430748 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:24:38.430926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.430764 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:24:38.430926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.430865 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:24:38.475876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.467792 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.475876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.468041 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.475876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.468079 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87v6j\" (UniqueName: \"kubernetes.io/projected/92a22e53-2e40-4024-a602-12f0c16918a0-kube-api-access-87v6j\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.569502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.569433 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.569502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.569498 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87v6j\" (UniqueName: \"kubernetes.io/projected/92a22e53-2e40-4024-a602-12f0c16918a0-kube-api-access-87v6j\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.569781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.569597 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.570330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.570259 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.571983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.571961 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.627906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.627839 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87v6j\" (UniqueName: \"kubernetes.io/projected/92a22e53-2e40-4024-a602-12f0c16918a0-kube-api-access-87v6j\") pod \"maas-api-revoke-sub-3ca8bf32-6ltzb\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:38.643132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:38.643105 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:39.021509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:39.021461 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb"] Oct 31 15:24:39.022230 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:39.022204 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a22e53_2e40_4024_a602_12f0c16918a0.slice/crio-c90c6c48dd84e9957f9fa7949f07e605f777a909155ff3311b22d086c5a83009 WatchSource:0}: Error finding container c90c6c48dd84e9957f9fa7949f07e605f777a909155ff3311b22d086c5a83009: Status 404 returned error can't find the container with id c90c6c48dd84e9957f9fa7949f07e605f777a909155ff3311b22d086c5a83009 Oct 31 15:24:39.136103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:39.136075 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" event={"ID":"92a22e53-2e40-4024-a602-12f0c16918a0","Type":"ContainerStarted","Data":"c90c6c48dd84e9957f9fa7949f07e605f777a909155ff3311b22d086c5a83009"} Oct 31 15:24:40.060069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:40.060035 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:40.177492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:40.176719 4865 generic.go:356] "Generic (PLEG): container finished" podID="92a22e53-2e40-4024-a602-12f0c16918a0" containerID="d229f63e55d558b3e3a62a57708e4e27ec8346c079140309c133651ca7e8e67c" exitCode=0 Oct 31 15:24:40.177492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:40.177249 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" event={"ID":"92a22e53-2e40-4024-a602-12f0c16918a0","Type":"ContainerDied","Data":"d229f63e55d558b3e3a62a57708e4e27ec8346c079140309c133651ca7e8e67c"} Oct 31 15:24:40.218955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:40.218922 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:40.369099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:40.369068 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:41.351898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.351870 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:41.427561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.427535 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a22e53-2e40-4024-a602-12f0c16918a0-kube-api-access-87v6j\" (UniqueName: \"kubernetes.io/projected/92a22e53-2e40-4024-a602-12f0c16918a0-kube-api-access-87v6j\") pod \"92a22e53-2e40-4024-a602-12f0c16918a0\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " Oct 31 15:24:41.427695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.427652 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-openshift-ca\") pod \"92a22e53-2e40-4024-a602-12f0c16918a0\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " Oct 31 15:24:41.427750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.427701 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-xks-ca\") pod \"92a22e53-2e40-4024-a602-12f0c16918a0\" (UID: \"92a22e53-2e40-4024-a602-12f0c16918a0\") " Oct 31 15:24:41.428051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.428024 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-openshift-ca" pod "92a22e53-2e40-4024-a602-12f0c16918a0" (UID: "92a22e53-2e40-4024-a602-12f0c16918a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:41.428216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.428201 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:41.429832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.429801 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a22e53-2e40-4024-a602-12f0c16918a0-kube-api-access-87v6j" pod "92a22e53-2e40-4024-a602-12f0c16918a0" (UID: "92a22e53-2e40-4024-a602-12f0c16918a0"). InnerVolumeSpecName "kube-api-access-87v6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:41.429832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.429810 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-xks-ca" pod "92a22e53-2e40-4024-a602-12f0c16918a0" (UID: "92a22e53-2e40-4024-a602-12f0c16918a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:24:41.529381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.529352 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a22e53-2e40-4024-a602-12f0c16918a0-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:41.529381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:41.529379 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87v6j\" (UniqueName: \"kubernetes.io/projected/92a22e53-2e40-4024-a602-12f0c16918a0-kube-api-access-87v6j\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:42.097953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.097922 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f"] Oct 31 15:24:42.098541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.098527 4865 cpu_manager.go:414] "containerMap: removing container" podUID="92a22e53-2e40-4024-a602-12f0c16918a0" containerName="revoke-keys" Oct 31 15:24:42.098600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.098543 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a22e53-2e40-4024-a602-12f0c16918a0" containerName="revoke-keys" Oct 31 15:24:42.098650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.098640 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a22e53-2e40-4024-a602-12f0c16918a0" containerName="revoke-keys" Oct 31 15:24:42.102291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.102276 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.242839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.242799 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcq6d\" (UniqueName: \"kubernetes.io/projected/f74dd8be-2cad-4699-b645-a4f905e5feca-kube-api-access-wcq6d\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.243137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.243104 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.243426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.243398 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.244971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.244943 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f"] Oct 31 15:24:42.251105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.251076 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" Oct 31 15:24:42.251215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.251124 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb" event={"ID":"92a22e53-2e40-4024-a602-12f0c16918a0","Type":"ContainerDied","Data":"c90c6c48dd84e9957f9fa7949f07e605f777a909155ff3311b22d086c5a83009"} Oct 31 15:24:42.251215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.251168 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c90c6c48dd84e9957f9fa7949f07e605f777a909155ff3311b22d086c5a83009" Oct 31 15:24:42.345044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.345017 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.345221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.345069 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.345221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.345124 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcq6d\" (UniqueName: \"kubernetes.io/projected/f74dd8be-2cad-4699-b645-a4f905e5feca-kube-api-access-wcq6d\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.345724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.345705 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.347633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.347613 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.403220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.403150 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcq6d\" (UniqueName: \"kubernetes.io/projected/f74dd8be-2cad-4699-b645-a4f905e5feca-kube-api-access-wcq6d\") pod \"maas-api-revoke-sub-3b788651-h7j9f\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.415945 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.415923 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:42.789585 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:42.789542 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74dd8be_2cad_4699_b645_a4f905e5feca.slice/crio-d06b4191dca7af83e9c7f3bfb6d63bbdf4cf4543858769171f7a85df3b199fd5 WatchSource:0}: Error finding container d06b4191dca7af83e9c7f3bfb6d63bbdf4cf4543858769171f7a85df3b199fd5: Status 404 returned error can't find the container with id d06b4191dca7af83e9c7f3bfb6d63bbdf4cf4543858769171f7a85df3b199fd5 Oct 31 15:24:42.801488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:42.793837 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f"] Oct 31 15:24:43.259357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:43.259322 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" event={"ID":"f74dd8be-2cad-4699-b645-a4f905e5feca","Type":"ContainerStarted","Data":"948f58e116ec91c94af4462ac6df9e2f064fa62b42ee54286c95cacce19765a0"} Oct 31 15:24:43.259523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:43.259385 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" event={"ID":"f74dd8be-2cad-4699-b645-a4f905e5feca","Type":"ContainerStarted","Data":"d06b4191dca7af83e9c7f3bfb6d63bbdf4cf4543858769171f7a85df3b199fd5"} Oct 31 15:24:43.377571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:43.377484 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" podStartSLOduration=1.37745885 podStartE2EDuration="1.37745885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:43.375357571 +0000 UTC m=+2328.994908984" watchObservedRunningTime="2026-10-31 15:24:43.37745885 +0000 UTC m=+2328.997010304" Oct 31 15:24:44.268677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:44.268639 4865 generic.go:356] "Generic (PLEG): container finished" podID="f74dd8be-2cad-4699-b645-a4f905e5feca" containerID="948f58e116ec91c94af4462ac6df9e2f064fa62b42ee54286c95cacce19765a0" exitCode=0 Oct 31 15:24:44.269092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:44.268714 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" event={"ID":"f74dd8be-2cad-4699-b645-a4f905e5feca","Type":"ContainerDied","Data":"948f58e116ec91c94af4462ac6df9e2f064fa62b42ee54286c95cacce19765a0"} Oct 31 15:24:44.986374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:44.986322 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:45.450499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.450457 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:45.498241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.498209 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-xks-ca\") pod \"f74dd8be-2cad-4699-b645-a4f905e5feca\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " Oct 31 15:24:45.498416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.498300 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74dd8be-2cad-4699-b645-a4f905e5feca-kube-api-access-wcq6d\" (UniqueName: \"kubernetes.io/projected/f74dd8be-2cad-4699-b645-a4f905e5feca-kube-api-access-wcq6d\") pod \"f74dd8be-2cad-4699-b645-a4f905e5feca\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " Oct 31 15:24:45.498523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.498501 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-openshift-ca\") pod \"f74dd8be-2cad-4699-b645-a4f905e5feca\" (UID: \"f74dd8be-2cad-4699-b645-a4f905e5feca\") " Oct 31 15:24:45.498853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.498822 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-openshift-ca" pod "f74dd8be-2cad-4699-b645-a4f905e5feca" (UID: "f74dd8be-2cad-4699-b645-a4f905e5feca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:45.499339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.499316 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:45.500310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.500287 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-xks-ca" pod "f74dd8be-2cad-4699-b645-a4f905e5feca" (UID: "f74dd8be-2cad-4699-b645-a4f905e5feca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:24:45.500546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.500522 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74dd8be-2cad-4699-b645-a4f905e5feca-kube-api-access-wcq6d" pod "f74dd8be-2cad-4699-b645-a4f905e5feca" (UID: "f74dd8be-2cad-4699-b645-a4f905e5feca"). InnerVolumeSpecName "kube-api-access-wcq6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:45.600189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.600099 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74dd8be-2cad-4699-b645-a4f905e5feca-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:45.600189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:45.600134 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcq6d\" (UniqueName: \"kubernetes.io/projected/f74dd8be-2cad-4699-b645-a4f905e5feca-kube-api-access-wcq6d\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:46.283636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:46.283603 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" Oct 31 15:24:46.283828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:46.283641 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f" event={"ID":"f74dd8be-2cad-4699-b645-a4f905e5feca","Type":"ContainerDied","Data":"d06b4191dca7af83e9c7f3bfb6d63bbdf4cf4543858769171f7a85df3b199fd5"} Oct 31 15:24:46.283828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:46.283685 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d06b4191dca7af83e9c7f3bfb6d63bbdf4cf4543858769171f7a85df3b199fd5" Oct 31 15:24:47.491375 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:24:47.491342 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:24:49.685943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:49.685909 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:49.820995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:49.820965 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:49.941158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:49.941068 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:52.777261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:52.777224 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:24:54.570227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:54.570189 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:56.118622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:56.118569 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:56.119638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:56.119608 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:24:56.155507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:56.154869 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:24:56.156747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:56.155812 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:24:56.216507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:56.204005 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:24:56.257940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:56.257895 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:56.505545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:56.504795 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:57.015287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.015247 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f"] Oct 31 15:24:57.018586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.018558 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f74dd8be-2cad-4699-b645-a4f905e5feca" containerName="revoke-keys" Oct 31 15:24:57.018586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.018584 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74dd8be-2cad-4699-b645-a4f905e5feca" containerName="revoke-keys" Oct 31 15:24:57.018797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.018767 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74dd8be-2cad-4699-b645-a4f905e5feca" containerName="revoke-keys" Oct 31 15:24:57.023858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.023838 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.067589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.067563 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f"] Oct 31 15:24:57.089186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.089159 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.089354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.089273 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsnkt\" (UniqueName: \"kubernetes.io/projected/54ff8b15-0993-48b7-aa54-137382b5c198-kube-api-access-jsnkt\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.089354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.089306 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.118821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.118791 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:24:57.119315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.118791 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:24:57.119404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.118883 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:24:57.128398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.128378 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:24:57.190778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.190745 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsnkt\" (UniqueName: \"kubernetes.io/projected/54ff8b15-0993-48b7-aa54-137382b5c198-kube-api-access-jsnkt\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.190937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.190785 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.191008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.190967 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.191464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.191444 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.193376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.193358 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.248619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.248593 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsnkt\" (UniqueName: \"kubernetes.io/projected/54ff8b15-0993-48b7-aa54-137382b5c198-kube-api-access-jsnkt\") pod \"maas-api-revoke-sub-512a45ea-hzn2f\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.338039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.337957 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:24:57.726074 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:24:57.726045 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ff8b15_0993_48b7_aa54_137382b5c198.slice/crio-e41d60a4ad5077dd69f9cfce3258c43f3796f0e1f7e25d30a9bb00f3be88051d WatchSource:0}: Error finding container e41d60a4ad5077dd69f9cfce3258c43f3796f0e1f7e25d30a9bb00f3be88051d: Status 404 returned error can't find the container with id e41d60a4ad5077dd69f9cfce3258c43f3796f0e1f7e25d30a9bb00f3be88051d Oct 31 15:24:57.726728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:57.726706 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f"] Oct 31 15:24:58.045950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:58.045919 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:58.457008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:58.456921 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" event={"ID":"54ff8b15-0993-48b7-aa54-137382b5c198","Type":"ContainerStarted","Data":"8c46a3b6fe7cf0a2db5681d250dd43eb45e7177f116be3f900f427e0190056f9"} Oct 31 15:24:58.457377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:58.457014 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" event={"ID":"54ff8b15-0993-48b7-aa54-137382b5c198","Type":"ContainerStarted","Data":"e41d60a4ad5077dd69f9cfce3258c43f3796f0e1f7e25d30a9bb00f3be88051d"} Oct 31 15:24:58.586994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:58.586930 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" podStartSLOduration=2.586914873 podStartE2EDuration="2.586914873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:58.58267365 +0000 UTC m=+2344.202225071" watchObservedRunningTime="2026-10-31 15:24:58.586914873 +0000 UTC m=+2344.206466283" Oct 31 15:24:59.465928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:59.465896 4865 generic.go:356] "Generic (PLEG): container finished" podID="54ff8b15-0993-48b7-aa54-137382b5c198" containerID="8c46a3b6fe7cf0a2db5681d250dd43eb45e7177f116be3f900f427e0190056f9" exitCode=0 Oct 31 15:24:59.466422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:59.465987 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" event={"ID":"54ff8b15-0993-48b7-aa54-137382b5c198","Type":"ContainerDied","Data":"8c46a3b6fe7cf0a2db5681d250dd43eb45e7177f116be3f900f427e0190056f9"} Oct 31 15:24:59.508043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:59.507981 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:24:59.635053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:24:59.635018 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:00.636925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.636888 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:25:00.662815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.662783 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ff8b15-0993-48b7-aa54-137382b5c198-kube-api-access-jsnkt\" (UniqueName: \"kubernetes.io/projected/54ff8b15-0993-48b7-aa54-137382b5c198-kube-api-access-jsnkt\") pod \"54ff8b15-0993-48b7-aa54-137382b5c198\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " Oct 31 15:25:00.663001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.662826 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-openshift-ca\") pod \"54ff8b15-0993-48b7-aa54-137382b5c198\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " Oct 31 15:25:00.663001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.662990 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-xks-ca\") pod \"54ff8b15-0993-48b7-aa54-137382b5c198\" (UID: \"54ff8b15-0993-48b7-aa54-137382b5c198\") " Oct 31 15:25:00.663295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.663265 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-openshift-ca" pod "54ff8b15-0993-48b7-aa54-137382b5c198" (UID: "54ff8b15-0993-48b7-aa54-137382b5c198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:00.663665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.663636 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:00.665154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.665130 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-xks-ca" pod "54ff8b15-0993-48b7-aa54-137382b5c198" (UID: "54ff8b15-0993-48b7-aa54-137382b5c198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:00.665262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.665189 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ff8b15-0993-48b7-aa54-137382b5c198-kube-api-access-jsnkt" pod "54ff8b15-0993-48b7-aa54-137382b5c198" (UID: "54ff8b15-0993-48b7-aa54-137382b5c198"). InnerVolumeSpecName "kube-api-access-jsnkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:00.765647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.765559 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsnkt\" (UniqueName: \"kubernetes.io/projected/54ff8b15-0993-48b7-aa54-137382b5c198-kube-api-access-jsnkt\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:00.765647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:00.765601 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ff8b15-0993-48b7-aa54-137382b5c198-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:01.483550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:01.483523 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" Oct 31 15:25:01.483732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:01.483562 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f" event={"ID":"54ff8b15-0993-48b7-aa54-137382b5c198","Type":"ContainerDied","Data":"e41d60a4ad5077dd69f9cfce3258c43f3796f0e1f7e25d30a9bb00f3be88051d"} Oct 31 15:25:01.483732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:01.483611 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e41d60a4ad5077dd69f9cfce3258c43f3796f0e1f7e25d30a9bb00f3be88051d" Oct 31 15:25:03.500791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:03.500762 4865 generic.go:356] "Generic (PLEG): container finished" podID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerID="9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd" exitCode=0 Oct 31 15:25:03.501236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:03.500835 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" event={"ID":"0f131f3a-4de2-42b9-b875-f0d007a674fe","Type":"ContainerDied","Data":"9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd"} Oct 31 15:25:03.501236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:03.501217 4865 scope.go:122] "RemoveContainer" containerID="9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd" Oct 31 15:25:03.502718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:03.502693 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:03.502803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:03.502759 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:03.502803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:03.502795 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:03.591262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:03.591234 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:25:04.217500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:04.201813 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:04.527996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:04.527899 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" event={"ID":"0f131f3a-4de2-42b9-b875-f0d007a674fe","Type":"ContainerStarted","Data":"03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24"} Oct 31 15:25:04.572151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:04.572115 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:25:04.573529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:04.573505 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 15:25:04.573668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:04.573560 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 15:25:05.572096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:05.572067 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 15:25:05.572539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:05.572115 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 15:25:06.067053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.067018 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r"] Oct 31 15:25:06.067695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.067680 4865 cpu_manager.go:414] "containerMap: removing container" podUID="54ff8b15-0993-48b7-aa54-137382b5c198" containerName="revoke-keys" Oct 31 15:25:06.067751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.067697 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ff8b15-0993-48b7-aa54-137382b5c198" containerName="revoke-keys" Oct 31 15:25:06.067829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.067820 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ff8b15-0993-48b7-aa54-137382b5c198" containerName="revoke-keys" Oct 31 15:25:06.071416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.071400 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.123312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.123279 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8"] Oct 31 15:25:06.127886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.127866 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.146942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.146918 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.146963 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.147014 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.147048 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.147090 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.147115 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.147146 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.147196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.147280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.147241 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njggk\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-kube-api-access-njggk\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.175849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.175819 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r"] Oct 31 15:25:06.231739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.231705 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8"] Oct 31 15:25:06.248743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248712 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njggk\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-kube-api-access-njggk\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.248919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248752 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.248919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248802 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.248919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248822 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.248919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248878 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.248919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248894 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.248919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248915 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.249273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248941 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.249273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.248956 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.249412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249393 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.249524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249430 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrqq\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-kube-api-access-pqrqq\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.249524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249502 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.249652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249527 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.249652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249550 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.249652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249586 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.249652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249607 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.249652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249611 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.249935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249665 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.249935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249682 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.249935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.249930 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.250103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.250085 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.250578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.250558 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.250731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.250692 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.251938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.251916 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.255935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.255914 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.326126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.326061 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.326313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.326061 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njggk\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-kube-api-access-njggk\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-4n56r\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.350644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.350620 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.350867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.350852 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.351189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.351173 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.351832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.351805 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.352264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.352249 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.351751 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.352572 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrqq\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-kube-api-access-pqrqq\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.351128 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.350936 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.352680 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.352736 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.352784 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.352964 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.353167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.353032 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.354308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.354274 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.354965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.354936 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.386153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.386132 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:06.420840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.420803 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.422174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.422154 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrqq\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-kube-api-access-pqrqq\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-qsjv8\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.444087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.443969 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:06.573034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.571960 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 15:25:06.573034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.572018 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 15:25:06.846444 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:25:06.846413 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2bf269_a9df_49d5_a4e2_ba7efdabf735.slice/crio-26cc3c988b74f2310c6f0342c06968b11162f44390e6a7fcbe1715647c28e82b WatchSource:0}: Error finding container 26cc3c988b74f2310c6f0342c06968b11162f44390e6a7fcbe1715647c28e82b: Status 404 returned error can't find the container with id 26cc3c988b74f2310c6f0342c06968b11162f44390e6a7fcbe1715647c28e82b Oct 31 15:25:06.846985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.846962 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r"] Oct 31 15:25:06.849237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.849206 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:06.849312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.849270 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:06.849312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.849302 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:06.923806 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:25:06.923778 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbebde03_06ea_43c3_afff_a2cfbb4b4a3c.slice/crio-9e36d80884faedbee4e264f2070e3a0625f1b99be66f02ec4f8ffe909190d7c3 WatchSource:0}: Error finding container 9e36d80884faedbee4e264f2070e3a0625f1b99be66f02ec4f8ffe909190d7c3: Status 404 returned error can't find the container with id 9e36d80884faedbee4e264f2070e3a0625f1b99be66f02ec4f8ffe909190d7c3 Oct 31 15:25:06.925438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.925370 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8"] Oct 31 15:25:06.926379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.926344 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:06.926486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.926411 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:06.926486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:06.926441 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:07.564118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.564081 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" event={"ID":"af2bf269-a9df-49d5-a4e2-ba7efdabf735","Type":"ContainerStarted","Data":"64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6"} Oct 31 15:25:07.564323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.564159 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" event={"ID":"af2bf269-a9df-49d5-a4e2-ba7efdabf735","Type":"ContainerStarted","Data":"26cc3c988b74f2310c6f0342c06968b11162f44390e6a7fcbe1715647c28e82b"} Oct 31 15:25:07.566551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.566518 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" event={"ID":"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c","Type":"ContainerStarted","Data":"7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6"} Oct 31 15:25:07.566823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.566776 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" event={"ID":"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c","Type":"ContainerStarted","Data":"9e36d80884faedbee4e264f2070e3a0625f1b99be66f02ec4f8ffe909190d7c3"} Oct 31 15:25:07.572149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.572124 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 15:25:07.572271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.572173 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 15:25:07.687322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.687269 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podStartSLOduration=1.6872550670000002 podStartE2EDuration="1.687255067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:07.686814362 +0000 UTC m=+2353.306365785" watchObservedRunningTime="2026-10-31 15:25:07.687255067 +0000 UTC m=+2353.306806477" Oct 31 15:25:07.827550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:07.827373 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podStartSLOduration=1.8273507599999999 podStartE2EDuration="1.82735076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:07.813661623 +0000 UTC m=+2353.433213046" watchObservedRunningTime="2026-10-31 15:25:07.82735076 +0000 UTC m=+2353.446902173" Oct 31 15:25:08.386981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.386947 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:08.388320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.388296 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4n56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 31 15:25:08.388534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.388512 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 31 15:25:08.452497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.451848 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:08.456492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.455601 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 31 15:25:08.456492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.455665 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 31 15:25:08.580498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.579197 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 15:25:08.580498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.579268 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 15:25:08.940495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:08.939460 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:09.387143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:09.387098 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4n56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 31 15:25:09.387359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:09.387160 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 31 15:25:09.446454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:09.446415 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 31 15:25:09.446782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:09.446756 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 31 15:25:09.572129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:09.572085 4865 patch_prober.go:29] interesting pod/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 15:25:09.572376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:09.572355 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 15:25:10.386974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:10.386928 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4n56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 31 15:25:10.387462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:10.386992 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 31 15:25:10.445902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:10.445864 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 31 15:25:10.446085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:10.445928 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 31 15:25:10.576292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:10.576264 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:25:10.577024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:10.576999 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:25:10.577532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:10.577515 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:25:11.386677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:11.386647 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4n56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 31 15:25:11.386977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:11.386944 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 31 15:25:11.446560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:11.446525 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 31 15:25:11.446740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:11.446576 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 31 15:25:12.386897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.386863 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4n56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 31 15:25:12.387090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.386912 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 31 15:25:12.446385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.446353 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 31 15:25:12.446595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.446403 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 31 15:25:12.548065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.548025 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97"] Oct 31 15:25:12.553540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.553521 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.581805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.581779 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.581962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.581940 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.582073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.582049 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsq5\" (UniqueName: \"kubernetes.io/projected/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-kube-api-access-ffsq5\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.617544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.617512 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:25:12.621363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.621338 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97"] Oct 31 15:25:12.654877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.654792 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:25:12.655180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.655165 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:25:12.655370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.655334 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:25:12.683767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.683727 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.683951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.683903 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.684031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.683999 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsq5\" (UniqueName: \"kubernetes.io/projected/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-kube-api-access-ffsq5\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.684974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.684940 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.749292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.749255 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.750977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.750951 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsq5\" (UniqueName: \"kubernetes.io/projected/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-kube-api-access-ffsq5\") pod \"maas-api-revoke-sub-b238505e-j2m97\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:12.868731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:12.868698 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:13.294291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.294257 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97"] Oct 31 15:25:13.294867 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:25:13.294840 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78f3bd9_328d_42eb_8a3a_e87ba6e2f9fd.slice/crio-979d7390b6d52b44fb5a16c10743338bc57e21980dd131ce78fac07c4c7ecf29 WatchSource:0}: Error finding container 979d7390b6d52b44fb5a16c10743338bc57e21980dd131ce78fac07c4c7ecf29: Status 404 returned error can't find the container with id 979d7390b6d52b44fb5a16c10743338bc57e21980dd131ce78fac07c4c7ecf29 Oct 31 15:25:13.387418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.387374 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4n56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 31 15:25:13.387927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.387440 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 31 15:25:13.445587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.445548 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 31 15:25:13.445785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.445597 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 31 15:25:13.678215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.678172 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" event={"ID":"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd","Type":"ContainerStarted","Data":"26434c9969978554ce2f0c6f91e89cc1f18e7bf09b124d5362fc7d5b7d479cf5"} Oct 31 15:25:13.678435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.678234 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" event={"ID":"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd","Type":"ContainerStarted","Data":"979d7390b6d52b44fb5a16c10743338bc57e21980dd131ce78fac07c4c7ecf29"} Oct 31 15:25:13.798920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.798858 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" podStartSLOduration=1.798838892 podStartE2EDuration="1.798838892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:13.792675811 +0000 UTC m=+2359.412227223" watchObservedRunningTime="2026-10-31 15:25:13.798838892 +0000 UTC m=+2359.418390304" Oct 31 15:25:13.853911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:13.853837 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:14.689386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:14.689265 4865 generic.go:356] "Generic (PLEG): container finished" podID="f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" containerID="26434c9969978554ce2f0c6f91e89cc1f18e7bf09b124d5362fc7d5b7d479cf5" exitCode=0 Oct 31 15:25:14.689906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:14.689574 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" event={"ID":"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd","Type":"ContainerDied","Data":"26434c9969978554ce2f0c6f91e89cc1f18e7bf09b124d5362fc7d5b7d479cf5"} Oct 31 15:25:15.064812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.064775 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:15.066940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.066913 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:15.068814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.068779 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:25:15.081402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.081372 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:15.081793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.081777 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:15.085488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.082575 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:25:15.849305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.849280 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:15.932148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.932114 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-openshift-ca\") pod \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " Oct 31 15:25:15.932315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.932189 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-kube-api-access-ffsq5\" (UniqueName: \"kubernetes.io/projected/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-kube-api-access-ffsq5\") pod \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " Oct 31 15:25:15.932360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.932348 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-xks-ca\") pod \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\" (UID: \"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd\") " Oct 31 15:25:15.932541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.932511 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-openshift-ca" pod "f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" (UID: "f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:15.932837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.932817 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:15.934522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.934501 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-xks-ca" pod "f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" (UID: "f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:15.934611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:15.934528 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-kube-api-access-ffsq5" pod "f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" (UID: "f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd"). InnerVolumeSpecName "kube-api-access-ffsq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:16.034126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:16.034089 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:16.034126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:16.034120 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffsq5\" (UniqueName: \"kubernetes.io/projected/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd-kube-api-access-ffsq5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:16.704870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:16.704845 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" Oct 31 15:25:16.705037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:16.704889 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97" event={"ID":"f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd","Type":"ContainerDied","Data":"979d7390b6d52b44fb5a16c10743338bc57e21980dd131ce78fac07c4c7ecf29"} Oct 31 15:25:16.705037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:16.704936 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979d7390b6d52b44fb5a16c10743338bc57e21980dd131ce78fac07c4c7ecf29" Oct 31 15:25:17.490552 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:25:17.490512 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:25:20.323672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:20.323643 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:24.157300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:24.157262 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:25:25.095032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:25.094989 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:27.138495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:27.125548 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:25:27.138495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:27.125951 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:25:27.166496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:27.156616 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:27.166496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:27.157500 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:25:27.207506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:27.204886 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:27.264076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:27.264030 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:27.490596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:27.490554 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:29.056850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.056819 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f"] Oct 31 15:25:29.057515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.057498 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" containerName="revoke-keys" Oct 31 15:25:29.057573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.057516 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" containerName="revoke-keys" Oct 31 15:25:29.057635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.057626 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" containerName="revoke-keys" Oct 31 15:25:29.061891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.061867 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.116554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.116525 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f"] Oct 31 15:25:29.162257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.162224 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:25:29.162496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.162483 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:25:29.166046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.166028 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:25:29.166297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.166278 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:25:29.205127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.205074 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.205127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.205133 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/1dc275da-82c5-4415-9e02-82fb98f8d9cc-kube-api-access-4hm6f\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.205433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.205413 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.306779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.306744 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.306997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.306788 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/1dc275da-82c5-4415-9e02-82fb98f8d9cc-kube-api-access-4hm6f\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.307065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.306992 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.307447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.307425 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.309525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.309500 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.364879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.364850 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/1dc275da-82c5-4415-9e02-82fb98f8d9cc-kube-api-access-4hm6f\") pod \"maas-api-revoke-sub-482fd989-vnr4f\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.378739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.378716 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:29.758211 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:25:29.758176 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc275da_82c5_4415_9e02_82fb98f8d9cc.slice/crio-99011cf670e8576822480d72fa7327c93c7e89ea7b03de6584321d44f906ece0 WatchSource:0}: Error finding container 99011cf670e8576822480d72fa7327c93c7e89ea7b03de6584321d44f906ece0: Status 404 returned error can't find the container with id 99011cf670e8576822480d72fa7327c93c7e89ea7b03de6584321d44f906ece0 Oct 31 15:25:29.759509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.759460 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f"] Oct 31 15:25:29.885983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.885946 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" event={"ID":"1dc275da-82c5-4415-9e02-82fb98f8d9cc","Type":"ContainerStarted","Data":"99011cf670e8576822480d72fa7327c93c7e89ea7b03de6584321d44f906ece0"} Oct 31 15:25:29.962667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.962636 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj"] Oct 31 15:25:29.967941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:29.967923 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.023657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.023628 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj"] Oct 31 15:25:30.124853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.124821 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44kb\" (UniqueName: \"kubernetes.io/projected/619165bb-95ff-4afe-aa14-13b399874546-kube-api-access-l44kb\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.125234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.124872 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619165bb-95ff-4afe-aa14-13b399874546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.125234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.125076 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619165bb-95ff-4afe-aa14-13b399874546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.226414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.226385 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44kb\" (UniqueName: \"kubernetes.io/projected/619165bb-95ff-4afe-aa14-13b399874546-kube-api-access-l44kb\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.226586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.226423 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619165bb-95ff-4afe-aa14-13b399874546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.226656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.226580 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619165bb-95ff-4afe-aa14-13b399874546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.227217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.227198 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619165bb-95ff-4afe-aa14-13b399874546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.228875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.228855 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619165bb-95ff-4afe-aa14-13b399874546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.284174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.284103 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44kb\" (UniqueName: \"kubernetes.io/projected/619165bb-95ff-4afe-aa14-13b399874546-kube-api-access-l44kb\") pod \"maas-api-revoke-sub-7266c873-895pj\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.409198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.409170 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:30.582314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.582223 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:30.895215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.895138 4865 generic.go:356] "Generic (PLEG): container finished" podID="1dc275da-82c5-4415-9e02-82fb98f8d9cc" containerID="38686a550eb5373babf153ef8be022c25b51e03a4511e487c875d02a6ff67ae5" exitCode=0 Oct 31 15:25:30.895360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.895221 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" event={"ID":"1dc275da-82c5-4415-9e02-82fb98f8d9cc","Type":"ContainerDied","Data":"38686a550eb5373babf153ef8be022c25b51e03a4511e487c875d02a6ff67ae5"} Oct 31 15:25:30.958566 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:25:30.958540 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619165bb_95ff_4afe_aa14_13b399874546.slice/crio-0447f12244f227479d5b60d2f9cdacf7fecfff9821f7046048d362b154adc3ef WatchSource:0}: Error finding container 0447f12244f227479d5b60d2f9cdacf7fecfff9821f7046048d362b154adc3ef: Status 404 returned error can't find the container with id 0447f12244f227479d5b60d2f9cdacf7fecfff9821f7046048d362b154adc3ef Oct 31 15:25:30.959791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:30.959772 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj"] Oct 31 15:25:31.903523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:31.903462 4865 generic.go:356] "Generic (PLEG): container finished" podID="619165bb-95ff-4afe-aa14-13b399874546" containerID="56c5dbb525a527c2110422666ada5d1940bd861798d74a37b43b693e293005de" exitCode=0 Oct 31 15:25:31.903934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:31.903558 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" event={"ID":"619165bb-95ff-4afe-aa14-13b399874546","Type":"ContainerDied","Data":"56c5dbb525a527c2110422666ada5d1940bd861798d74a37b43b693e293005de"} Oct 31 15:25:31.903934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:31.903611 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" event={"ID":"619165bb-95ff-4afe-aa14-13b399874546","Type":"ContainerStarted","Data":"0447f12244f227479d5b60d2f9cdacf7fecfff9821f7046048d362b154adc3ef"} Oct 31 15:25:32.058215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.058190 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:32.153413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.153328 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-openshift-ca\") pod \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " Oct 31 15:25:32.153579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.153423 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc275da-82c5-4415-9e02-82fb98f8d9cc-kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/1dc275da-82c5-4415-9e02-82fb98f8d9cc-kube-api-access-4hm6f\") pod \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " Oct 31 15:25:32.153579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.153459 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-xks-ca\") pod \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\" (UID: \"1dc275da-82c5-4415-9e02-82fb98f8d9cc\") " Oct 31 15:25:32.153706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.153682 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-openshift-ca" pod "1dc275da-82c5-4415-9e02-82fb98f8d9cc" (UID: "1dc275da-82c5-4415-9e02-82fb98f8d9cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:32.154082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.154057 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:32.155832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.155810 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-xks-ca" pod "1dc275da-82c5-4415-9e02-82fb98f8d9cc" (UID: "1dc275da-82c5-4415-9e02-82fb98f8d9cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:32.155959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.155941 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc275da-82c5-4415-9e02-82fb98f8d9cc-kube-api-access-4hm6f" pod "1dc275da-82c5-4415-9e02-82fb98f8d9cc" (UID: "1dc275da-82c5-4415-9e02-82fb98f8d9cc"). InnerVolumeSpecName "kube-api-access-4hm6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:32.255418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.255384 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/1dc275da-82c5-4415-9e02-82fb98f8d9cc-kube-api-access-4hm6f\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:32.255418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.255414 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc275da-82c5-4415-9e02-82fb98f8d9cc-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:32.913492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.913395 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" Oct 31 15:25:32.913492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.913438 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f" event={"ID":"1dc275da-82c5-4415-9e02-82fb98f8d9cc","Type":"ContainerDied","Data":"99011cf670e8576822480d72fa7327c93c7e89ea7b03de6584321d44f906ece0"} Oct 31 15:25:32.913984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:32.913502 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99011cf670e8576822480d72fa7327c93c7e89ea7b03de6584321d44f906ece0" Oct 31 15:25:33.062074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.062051 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:33.165585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.165517 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619165bb-95ff-4afe-aa14-13b399874546-kube-api-access-l44kb\" (UniqueName: \"kubernetes.io/projected/619165bb-95ff-4afe-aa14-13b399874546-kube-api-access-l44kb\") pod \"619165bb-95ff-4afe-aa14-13b399874546\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " Oct 31 15:25:33.165759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.165671 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619165bb-95ff-4afe-aa14-13b399874546-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619165bb-95ff-4afe-aa14-13b399874546-maas-api-openshift-ca\") pod \"619165bb-95ff-4afe-aa14-13b399874546\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " Oct 31 15:25:33.165759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.165731 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/619165bb-95ff-4afe-aa14-13b399874546-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619165bb-95ff-4afe-aa14-13b399874546-maas-api-xks-ca\") pod \"619165bb-95ff-4afe-aa14-13b399874546\" (UID: \"619165bb-95ff-4afe-aa14-13b399874546\") " Oct 31 15:25:33.166039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.166015 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619165bb-95ff-4afe-aa14-13b399874546-maas-api-openshift-ca" pod "619165bb-95ff-4afe-aa14-13b399874546" (UID: "619165bb-95ff-4afe-aa14-13b399874546"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:33.166299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.166272 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619165bb-95ff-4afe-aa14-13b399874546-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:33.167793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.167765 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619165bb-95ff-4afe-aa14-13b399874546-kube-api-access-l44kb" pod "619165bb-95ff-4afe-aa14-13b399874546" (UID: "619165bb-95ff-4afe-aa14-13b399874546"). InnerVolumeSpecName "kube-api-access-l44kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:33.167896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.167843 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619165bb-95ff-4afe-aa14-13b399874546-maas-api-xks-ca" pod "619165bb-95ff-4afe-aa14-13b399874546" (UID: "619165bb-95ff-4afe-aa14-13b399874546"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:33.269420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.269388 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44kb\" (UniqueName: \"kubernetes.io/projected/619165bb-95ff-4afe-aa14-13b399874546-kube-api-access-l44kb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:33.269420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.269419 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619165bb-95ff-4afe-aa14-13b399874546-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:33.921556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.921525 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" Oct 31 15:25:33.922057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.921521 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj" event={"ID":"619165bb-95ff-4afe-aa14-13b399874546","Type":"ContainerDied","Data":"0447f12244f227479d5b60d2f9cdacf7fecfff9821f7046048d362b154adc3ef"} Oct 31 15:25:33.922057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:33.921669 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0447f12244f227479d5b60d2f9cdacf7fecfff9821f7046048d362b154adc3ef" Oct 31 15:25:35.108211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:35.108180 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:36.316116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.316086 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78"] Oct 31 15:25:36.316910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.316888 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1dc275da-82c5-4415-9e02-82fb98f8d9cc" containerName="revoke-keys" Oct 31 15:25:36.317029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.316912 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc275da-82c5-4415-9e02-82fb98f8d9cc" containerName="revoke-keys" Oct 31 15:25:36.317029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.316929 4865 cpu_manager.go:414] "containerMap: removing container" podUID="619165bb-95ff-4afe-aa14-13b399874546" containerName="revoke-keys" Oct 31 15:25:36.317029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.316936 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619165bb-95ff-4afe-aa14-13b399874546" containerName="revoke-keys" Oct 31 15:25:36.317194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.317122 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc275da-82c5-4415-9e02-82fb98f8d9cc" containerName="revoke-keys" Oct 31 15:25:36.317194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.317138 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="619165bb-95ff-4afe-aa14-13b399874546" containerName="revoke-keys" Oct 31 15:25:36.320892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.320866 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.415801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.415745 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.415801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.415790 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.416130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.415931 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.416130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.415984 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.416130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.416117 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.416314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.416188 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.416314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.416243 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swz2d\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-kube-api-access-swz2d\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.416314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.416270 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.416314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.416295 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.432735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.432700 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78"] Oct 31 15:25:36.517392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517362 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.517603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517406 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.517603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517462 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.517603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517514 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.517603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517547 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swz2d\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-kube-api-access-swz2d\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.517603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517562 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.517603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517577 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.518005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517667 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.518005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517701 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.518005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517921 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.518005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.517984 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.518393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.518108 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.518393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.518186 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.518542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.518489 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.519943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.519916 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.520097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.520081 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.576160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.576102 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.576682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.576664 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swz2d\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-kube-api-access-swz2d\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dwv78\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:36.634465 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:36.634435 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:37.066610 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:25:37.066570 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1a7f85_00e6_4f94_b74b_af9cdd2367ba.slice/crio-11c8b424e7f3f8feb59ce2e55ef6b2b9a8309c4117c8f059e9155033188704dd WatchSource:0}: Error finding container 11c8b424e7f3f8feb59ce2e55ef6b2b9a8309c4117c8f059e9155033188704dd: Status 404 returned error can't find the container with id 11c8b424e7f3f8feb59ce2e55ef6b2b9a8309c4117c8f059e9155033188704dd Oct 31 15:25:37.066973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:37.066953 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78"] Oct 31 15:25:37.068905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:37.068874 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:37.068978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:37.068947 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:37.069017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:37.068981 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:25:37.962713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:37.962676 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" event={"ID":"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba","Type":"ContainerStarted","Data":"b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf"} Oct 31 15:25:37.963115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:37.962731 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" event={"ID":"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba","Type":"ContainerStarted","Data":"11c8b424e7f3f8feb59ce2e55ef6b2b9a8309c4117c8f059e9155033188704dd"} Oct 31 15:25:38.185537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:38.185462 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podStartSLOduration=2.185448386 podStartE2EDuration="2.185448386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:38.133429008 +0000 UTC m=+2383.752980418" watchObservedRunningTime="2026-10-31 15:25:38.185448386 +0000 UTC m=+2383.804999798" Oct 31 15:25:38.634777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:38.634742 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:38.635961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:38.635936 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dwv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 15:25:38.636086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:38.635991 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 15:25:39.634791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:39.634759 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dwv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 15:25:39.635296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:39.634820 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 15:25:39.881061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:39.881025 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:40.634909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:40.634879 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dwv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 15:25:40.635332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:40.634929 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 15:25:41.634865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:41.634831 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dwv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 15:25:41.635095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:41.634892 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 15:25:42.637686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:42.635741 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dwv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 15:25:42.637686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:42.635810 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 15:25:43.634786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:43.634750 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dwv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Oct 31 15:25:43.634971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:43.634812 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Oct 31 15:25:44.639493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.639444 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:44.641738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.641707 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq"] Oct 31 15:25:44.648036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.648017 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.737180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.729387 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq"] Oct 31 15:25:44.748272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.748246 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:25:44.748674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.748638 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:25:44.748912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.748886 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:25:44.758862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.758836 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.759075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.759049 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.759434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.759411 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m29r\" (UniqueName: \"kubernetes.io/projected/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-kube-api-access-5m29r\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.763623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.763602 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:25:44.863746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.863717 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.863930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.863757 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.863930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.863828 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m29r\" (UniqueName: \"kubernetes.io/projected/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-kube-api-access-5m29r\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.864393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.864361 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.866499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.866450 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.886941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.886916 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4"] Oct 31 15:25:44.887183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.887156 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" podUID="af15b77c-21ac-4120-9571-9729dbb6585f" containerName="discovery" containerID="cri-o://408eb18a9e2d127fd20e5fe042236ccb6ca5a91b8408d3633d806091c5cd3cec" gracePeriod=30 Oct 31 15:25:44.930221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.930143 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m29r\" (UniqueName: \"kubernetes.io/projected/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-kube-api-access-5m29r\") pod \"maas-api-revoke-sub-0d460577-mdxcq\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:44.997591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:44.986322 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:45.035044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.034623 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:45.037054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.035892 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:45.037916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.037893 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:25:45.069968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.069941 4865 generic.go:356] "Generic (PLEG): container finished" podID="af15b77c-21ac-4120-9571-9729dbb6585f" containerID="408eb18a9e2d127fd20e5fe042236ccb6ca5a91b8408d3633d806091c5cd3cec" exitCode=0 Oct 31 15:25:45.070093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.070009 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" event={"ID":"af15b77c-21ac-4120-9571-9729dbb6585f","Type":"ContainerDied","Data":"408eb18a9e2d127fd20e5fe042236ccb6ca5a91b8408d3633d806091c5cd3cec"} Oct 31 15:25:45.220879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.220854 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 15:25:45.370780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.370738 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-kube-api-access-579rw\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-kube-api-access-579rw\") pod \"af15b77c-21ac-4120-9571-9729dbb6585f\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " Oct 31 15:25:45.370983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.370872 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-ca-configmap\") pod \"af15b77c-21ac-4120-9571-9729dbb6585f\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " Oct 31 15:25:45.370983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.370921 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af15b77c-21ac-4120-9571-9729dbb6585f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af15b77c-21ac-4120-9571-9729dbb6585f-local-certs\") pod \"af15b77c-21ac-4120-9571-9729dbb6585f\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " Oct 31 15:25:45.370983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.370976 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-dns-cert\") pod \"af15b77c-21ac-4120-9571-9729dbb6585f\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " Oct 31 15:25:45.371171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.371118 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-kubeconfig\") pod \"af15b77c-21ac-4120-9571-9729dbb6585f\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " Oct 31 15:25:45.371228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.371170 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-istio-token\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-istio-token\") pod \"af15b77c-21ac-4120-9571-9729dbb6585f\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " Oct 31 15:25:45.371291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.371235 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-cacerts\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-cacerts\") pod \"af15b77c-21ac-4120-9571-9729dbb6585f\" (UID: \"af15b77c-21ac-4120-9571-9729dbb6585f\") " Oct 31 15:25:45.371291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.371237 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-ca-configmap" pod "af15b77c-21ac-4120-9571-9729dbb6585f" (UID: "af15b77c-21ac-4120-9571-9729dbb6585f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:45.372116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.372089 4865 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-ca-configmap\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:45.373625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.373592 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af15b77c-21ac-4120-9571-9729dbb6585f-local-certs" pod "af15b77c-21ac-4120-9571-9729dbb6585f" (UID: "af15b77c-21ac-4120-9571-9729dbb6585f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:25:45.373953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.373920 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-kubeconfig" pod "af15b77c-21ac-4120-9571-9729dbb6585f" (UID: "af15b77c-21ac-4120-9571-9729dbb6585f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:45.374097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.373956 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-dns-cert" pod "af15b77c-21ac-4120-9571-9729dbb6585f" (UID: "af15b77c-21ac-4120-9571-9729dbb6585f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:45.374097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.374034 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-kube-api-access-579rw" pod "af15b77c-21ac-4120-9571-9729dbb6585f" (UID: "af15b77c-21ac-4120-9571-9729dbb6585f"). InnerVolumeSpecName "kube-api-access-579rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:45.374197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.374127 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-istio-token" pod "af15b77c-21ac-4120-9571-9729dbb6585f" (UID: "af15b77c-21ac-4120-9571-9729dbb6585f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:45.374197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.374158 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-cacerts" pod "af15b77c-21ac-4120-9571-9729dbb6585f" (UID: "af15b77c-21ac-4120-9571-9729dbb6585f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:45.391995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.391830 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq"] Oct 31 15:25:45.473775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.473694 4865 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-csr-dns-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:45.473775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.473728 4865 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-istio-kubeconfig\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:45.473775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.473741 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:45.473775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.473754 4865 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af15b77c-21ac-4120-9571-9729dbb6585f-cacerts\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:45.473775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.473770 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-579rw\" (UniqueName: \"kubernetes.io/projected/af15b77c-21ac-4120-9571-9729dbb6585f-kube-api-access-579rw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:45.474105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:45.473785 4865 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af15b77c-21ac-4120-9571-9729dbb6585f-local-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:46.079545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.079511 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" event={"ID":"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff","Type":"ContainerStarted","Data":"d2aacda5778b719ca596b198994041102c6505f4cbed9b2051a93eaed5795f89"} Oct 31 15:25:46.079980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.079567 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" event={"ID":"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff","Type":"ContainerStarted","Data":"265a7201c1b3de3663ef443b71e1ad294d170674a2ab9fb2c2b6bab235145a35"} Oct 31 15:25:46.081210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.081188 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" Oct 31 15:25:46.081332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.081224 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4" event={"ID":"af15b77c-21ac-4120-9571-9729dbb6585f","Type":"ContainerDied","Data":"9e6acdae0c54e16162a69b73cd59bcf2f666d990ad80af3b6686ca914120205f"} Oct 31 15:25:46.081332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.081282 4865 scope.go:122] "RemoveContainer" containerID="408eb18a9e2d127fd20e5fe042236ccb6ca5a91b8408d3633d806091c5cd3cec" Oct 31 15:25:46.198507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.198370 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" podStartSLOduration=2.198355843 podStartE2EDuration="2.198355843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:46.195304839 +0000 UTC m=+2391.814856253" watchObservedRunningTime="2026-10-31 15:25:46.198355843 +0000 UTC m=+2391.817907291" Oct 31 15:25:46.312027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.312002 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4"] Oct 31 15:25:46.362555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.362525 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6n8v4"] Oct 31 15:25:46.499441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:46.499409 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af15b77c-21ac-4120-9571-9729dbb6585f" path="/var/lib/kubelet/pods/af15b77c-21ac-4120-9571-9729dbb6585f/volumes" Oct 31 15:25:47.089753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:47.089720 4865 generic.go:356] "Generic (PLEG): container finished" podID="6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" containerID="d2aacda5778b719ca596b198994041102c6505f4cbed9b2051a93eaed5795f89" exitCode=0 Oct 31 15:25:47.090209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:47.089806 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" event={"ID":"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff","Type":"ContainerDied","Data":"d2aacda5778b719ca596b198994041102c6505f4cbed9b2051a93eaed5795f89"} Oct 31 15:25:47.491056 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:25:47.491022 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:25:48.147022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.146982 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:48.289115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.289091 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:48.424806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.424709 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-xks-ca\") pod \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " Oct 31 15:25:48.424989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.424815 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-openshift-ca\") pod \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " Oct 31 15:25:48.424989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.424982 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-kube-api-access-5m29r\" (UniqueName: \"kubernetes.io/projected/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-kube-api-access-5m29r\") pod \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\" (UID: \"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff\") " Oct 31 15:25:48.425623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.425344 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-openshift-ca" pod "6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" (UID: "6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:48.426120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.426098 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:48.428982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.427856 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-xks-ca" pod "6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" (UID: "6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:48.430514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.430453 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-kube-api-access-5m29r" pod "6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" (UID: "6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff"). InnerVolumeSpecName "kube-api-access-5m29r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:48.527156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.527126 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:48.527291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.527171 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m29r\" (UniqueName: \"kubernetes.io/projected/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff-kube-api-access-5m29r\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:48.903986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:48.903956 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7"] Oct 31 15:25:49.010620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.010212 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln"] Oct 31 15:25:49.106982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.106949 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" Oct 31 15:25:49.107166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.107001 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq" event={"ID":"6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff","Type":"ContainerDied","Data":"265a7201c1b3de3663ef443b71e1ad294d170674a2ab9fb2c2b6bab235145a35"} Oct 31 15:25:49.107166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.107042 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="265a7201c1b3de3663ef443b71e1ad294d170674a2ab9fb2c2b6bab235145a35" Oct 31 15:25:49.288355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.288318 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:49.425185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.425149 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:49.602392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.602301 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:49.936109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.936008 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx"] Oct 31 15:25:49.936308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:49.936213 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" containerID="cri-o://b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc" gracePeriod=30 Oct 31 15:25:52.804062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.804020 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq"] Oct 31 15:25:52.804700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.804684 4865 cpu_manager.go:414] "containerMap: removing container" podUID="af15b77c-21ac-4120-9571-9729dbb6585f" containerName="discovery" Oct 31 15:25:52.804755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.804702 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af15b77c-21ac-4120-9571-9729dbb6585f" containerName="discovery" Oct 31 15:25:52.804755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.804738 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" containerName="revoke-keys" Oct 31 15:25:52.804755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.804743 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" containerName="revoke-keys" Oct 31 15:25:52.813416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.813379 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" containerName="revoke-keys" Oct 31 15:25:52.813598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.813428 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="af15b77c-21ac-4120-9571-9729dbb6585f" containerName="discovery" Oct 31 15:25:52.818526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.818502 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:52.863399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.863373 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq"] Oct 31 15:25:52.897882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.897852 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:52.898074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.897888 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:52.898074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.897934 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8h2x\" (UniqueName: \"kubernetes.io/projected/56877668-57fb-43d1-b731-96593a6c9b6e-kube-api-access-j8h2x\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:52.909614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.909591 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:25:52.909727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.909685 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:25:52.909727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.909698 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:25:52.923021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.922997 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:25:52.999663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.999632 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:52.999801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.999671 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:52.999801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:52.999726 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8h2x\" (UniqueName: \"kubernetes.io/projected/56877668-57fb-43d1-b731-96593a6c9b6e-kube-api-access-j8h2x\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:53.000374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:53.000357 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:53.002325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:53.002302 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:53.057320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:53.057247 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8h2x\" (UniqueName: \"kubernetes.io/projected/56877668-57fb-43d1-b731-96593a6c9b6e-kube-api-access-j8h2x\") pod \"maas-api-revoke-sub-f00d2ab5-sfzlq\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:53.131916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:53.131889 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:53.511407 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:25:53.511378 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56877668_57fb_43d1_b731_96593a6c9b6e.slice/crio-a174f38edc577ff9f2fc78f80f386b54a9609eb1d3a775996df55ba7eeb65eef WatchSource:0}: Error finding container a174f38edc577ff9f2fc78f80f386b54a9609eb1d3a775996df55ba7eeb65eef: Status 404 returned error can't find the container with id a174f38edc577ff9f2fc78f80f386b54a9609eb1d3a775996df55ba7eeb65eef Oct 31 15:25:53.512388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:53.512369 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq"] Oct 31 15:25:53.844178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:53.844088 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:53.905204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:53.905160 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" podUID="26c7b7ef-fdb4-48da-9155-0602fcdf0482" containerName="payload-processing" containerID="cri-o://a59ff0c60774026109f3cd4e3121bee92e379512eb3a809dc1565adc4bc3918c" gracePeriod=25 Oct 31 15:25:54.012052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.012002 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" podUID="fd991c02-0fa9-4719-b7c6-0078ba98b9e5" containerName="payload-pre-processing" containerID="cri-o://54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3" gracePeriod=25 Oct 31 15:25:54.178563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.176419 4865 generic.go:356] "Generic (PLEG): container finished" podID="fd991c02-0fa9-4719-b7c6-0078ba98b9e5" containerID="54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3" exitCode=0 Oct 31 15:25:54.178563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.176651 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" event={"ID":"fd991c02-0fa9-4719-b7c6-0078ba98b9e5","Type":"ContainerDied","Data":"54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3"} Oct 31 15:25:54.182785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.181867 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" event={"ID":"56877668-57fb-43d1-b731-96593a6c9b6e","Type":"ContainerStarted","Data":"be7089ebfe940f7149db79724d26162600aca93788551efef30e6ac9195c58d3"} Oct 31 15:25:54.182785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.181923 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" event={"ID":"56877668-57fb-43d1-b731-96593a6c9b6e","Type":"ContainerStarted","Data":"a174f38edc577ff9f2fc78f80f386b54a9609eb1d3a775996df55ba7eeb65eef"} Oct 31 15:25:54.186686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.186636 4865 generic.go:356] "Generic (PLEG): container finished" podID="26c7b7ef-fdb4-48da-9155-0602fcdf0482" containerID="a59ff0c60774026109f3cd4e3121bee92e379512eb3a809dc1565adc4bc3918c" exitCode=0 Oct 31 15:25:54.186912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.186843 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" event={"ID":"26c7b7ef-fdb4-48da-9155-0602fcdf0482","Type":"ContainerDied","Data":"a59ff0c60774026109f3cd4e3121bee92e379512eb3a809dc1565adc4bc3918c"} Oct 31 15:25:54.250743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.250656 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:25:54.395408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.334569 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c7b7ef-fdb4-48da-9155-0602fcdf0482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c7b7ef-fdb4-48da-9155-0602fcdf0482-plugins-config-volume\") pod \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " Oct 31 15:25:54.395408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.334705 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c7b7ef-fdb4-48da-9155-0602fcdf0482-kube-api-access-zsf6d\" (UniqueName: \"kubernetes.io/projected/26c7b7ef-fdb4-48da-9155-0602fcdf0482-kube-api-access-zsf6d\") pod \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\" (UID: \"26c7b7ef-fdb4-48da-9155-0602fcdf0482\") " Oct 31 15:25:54.395408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.371699 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c7b7ef-fdb4-48da-9155-0602fcdf0482-plugins-config-volume" pod "26c7b7ef-fdb4-48da-9155-0602fcdf0482" (UID: "26c7b7ef-fdb4-48da-9155-0602fcdf0482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:54.395408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.395310 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c7b7ef-fdb4-48da-9155-0602fcdf0482-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:54.434584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.432993 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:54.455003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.435103 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" podStartSLOduration=2.435085002 podStartE2EDuration="2.435085002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:54.329924084 +0000 UTC m=+2399.949475496" watchObservedRunningTime="2026-10-31 15:25:54.435085002 +0000 UTC m=+2400.054636413" Oct 31 15:25:54.467542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.458717 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c7b7ef-fdb4-48da-9155-0602fcdf0482-kube-api-access-zsf6d" pod "26c7b7ef-fdb4-48da-9155-0602fcdf0482" (UID: "26c7b7ef-fdb4-48da-9155-0602fcdf0482"). InnerVolumeSpecName "kube-api-access-zsf6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:54.504151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:54.500098 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsf6d\" (UniqueName: \"kubernetes.io/projected/26c7b7ef-fdb4-48da-9155-0602fcdf0482-kube-api-access-zsf6d\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:55.271933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.269807 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:25:55.304876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.287139 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" event={"ID":"fd991c02-0fa9-4719-b7c6-0078ba98b9e5","Type":"ContainerDied","Data":"24c92bf7c8e0d43ed036c892dda5b445a6c9e804d026ab18f9e3fadfef3db670"} Oct 31 15:25:55.304876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.287206 4865 scope.go:122] "RemoveContainer" containerID="54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3" Oct 31 15:25:55.356136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.356092 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-plugins-config-volume\") pod \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " Oct 31 15:25:55.356412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.356374 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-kube-api-access-2h44g\" (UniqueName: \"kubernetes.io/projected/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-kube-api-access-2h44g\") pod \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\" (UID: \"fd991c02-0fa9-4719-b7c6-0078ba98b9e5\") " Oct 31 15:25:55.362718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.362682 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-plugins-config-volume" pod "fd991c02-0fa9-4719-b7c6-0078ba98b9e5" (UID: "fd991c02-0fa9-4719-b7c6-0078ba98b9e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:55.428988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.428937 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-kube-api-access-2h44g" pod "fd991c02-0fa9-4719-b7c6-0078ba98b9e5" (UID: "fd991c02-0fa9-4719-b7c6-0078ba98b9e5"). InnerVolumeSpecName "kube-api-access-2h44g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:55.469523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.462806 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" Oct 31 15:25:55.469523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.463363 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7" event={"ID":"26c7b7ef-fdb4-48da-9155-0602fcdf0482","Type":"ContainerDied","Data":"a5658aba0750af995de5c0b04cb2ea84e41941de236b036a4a502e0e08d54960"} Oct 31 15:25:55.491434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.491399 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h44g\" (UniqueName: \"kubernetes.io/projected/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-kube-api-access-2h44g\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:55.522725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.522639 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd991c02-0fa9-4719-b7c6-0078ba98b9e5-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:55.814759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.814715 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:55.843429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.843397 4865 scope.go:122] "RemoveContainer" containerID="a59ff0c60774026109f3cd4e3121bee92e379512eb3a809dc1565adc4bc3918c" Oct 31 15:25:55.883496 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:25:55.864227 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3\": container with ID starting with 54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3 not found: ID does not exist" containerID="54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3" Oct 31 15:25:55.883864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.883831 4865 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3\": container with ID starting with 54184596970fa70e0145fd5e1ea3366c942003e271adf25bacb6092ddeb16ef3 not found: ID does not exist" Oct 31 15:25:55.884586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:55.880550 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.027771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.027709 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.099656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.099566 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7"] Oct 31 15:25:56.129696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.129638 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.130364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.130333 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:25:56.159597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.159552 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.160202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.160172 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:25:56.200503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.191977 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-76864498c7-s84g7"] Oct 31 15:25:56.204031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.203988 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.285090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.285046 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.585495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.538835 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.858548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.845137 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.895496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:56.894598 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:57.046382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.032805 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:25:57.098501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.098340 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c7b7ef-fdb4-48da-9155-0602fcdf0482" path="/var/lib/kubelet/pods/26c7b7ef-fdb4-48da-9155-0602fcdf0482/volumes" Oct 31 15:25:57.121601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.099198 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:25:57.121601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.099260 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:25:57.121601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.099297 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:25:57.134526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.131486 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:57.134526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.131781 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:25:57.134526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.132007 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:25:57.134526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.132062 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:25:57.166492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.163228 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:57.166492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.163333 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:25:57.183431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.182403 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:57.238086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.238031 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:57.257508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.254334 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:25:57.257508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.254401 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:25:57.257508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.254440 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:25:57.295625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.295577 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:57.304955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.300819 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:25:57.532553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.484238 4865 scope.go:122] "RemoveContainer" containerID="06d2e1c75f5ca973658e1a88d4530517397e5502170b699e418c536f88feb630" Oct 31 15:25:57.574712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.571242 4865 generic.go:356] "Generic (PLEG): container finished" podID="56877668-57fb-43d1-b731-96593a6c9b6e" containerID="be7089ebfe940f7149db79724d26162600aca93788551efef30e6ac9195c58d3" exitCode=0 Oct 31 15:25:57.575168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.575045 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" event={"ID":"56877668-57fb-43d1-b731-96593a6c9b6e","Type":"ContainerDied","Data":"be7089ebfe940f7149db79724d26162600aca93788551efef30e6ac9195c58d3"} Oct 31 15:25:57.631497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.622317 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:25:57.634720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.634694 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 15:25:57.682497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.660974 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:25:57.682497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.669882 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:25:57.825651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.825616 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-podinfo\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.916667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.916633 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-certs\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.916965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.916944 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-envoy\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.926589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.926560 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istiod-ca-cert\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.926803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.926786 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-token\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.926927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.926913 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-data\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.927131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.927111 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-credential-socket\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.927439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.927407 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-kube-api-access-jzzr5\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-kube-api-access-jzzr5\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.927658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.927635 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-socket\") pod \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\" (UID: \"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d\") " Oct 31 15:25:57.951495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.942239 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:25:57.991798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.991746 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-certs" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:25:57.991998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.897728 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:25:57.992147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.992127 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" event={"ID":"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d","Type":"ContainerDied","Data":"b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc"} Oct 31 15:25:57.992303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.992290 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:25:57.992404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.992395 4865 scope.go:122] "RemoveContainer" containerID="b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc" Oct 31 15:25:58.000843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.000802 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-data" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:25:58.001073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.001048 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-credential-socket" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:25:58.004524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:57.917220 4865 generic.go:356] "Generic (PLEG): container finished" podID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerID="b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc" exitCode=0 Oct 31 15:25:58.004524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.004153 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" event={"ID":"aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d","Type":"ContainerDied","Data":"284de8653d1152b4ccce7100dcd5e45f7b7b70d3eabb7d93bb42d81ff0728eeb"} Oct 31 15:25:58.012842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.012818 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:25:58.016978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.016958 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:25:58.027099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.026928 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-socket" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:25:58.027965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.027591 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:25:58.027965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.027648 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:25:58.029204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.028708 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:25:58.030437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.030415 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.030607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.030591 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.030706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.030692 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.030791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.030780 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.082930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.077191 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istiod-ca-cert" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:58.099223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.080627 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:25:58.099843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.099825 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln" Oct 31 15:25:58.116940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.116907 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-token" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:58.132050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.132023 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.132200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.132184 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.145384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.145358 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-podinfo" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:25:58.151181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.151158 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:25:58.172529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.172494 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-kube-api-access-jzzr5" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "kube-api-access-jzzr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:58.172857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.172824 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-envoy" pod "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" (UID: "aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:25:58.219272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.219249 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:25:58.228793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.228745 4865 scope.go:122] "RemoveContainer" containerID="b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc" Oct 31 15:25:58.230194 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:25:58.230163 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc\": container with ID starting with b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc not found: ID does not exist" containerID="b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc" Oct 31 15:25:58.230390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.230358 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc"} err="failed to get container status \"b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc\": rpc error: code = NotFound desc = could not find container \"b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc\": container with ID starting with b15b6c2761549188995eed9502758cec970c7ac9db6ca0c80fb2653f88ddcabc not found: ID does not exist" Oct 31 15:25:58.231577 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:25:58.231546 4865 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd991c02_0fa9_4719_b7c6_0078ba98b9e5.slice/crio-24c92bf7c8e0d43ed036c892dda5b445a6c9e804d026ab18f9e3fadfef3db670\": RecentStats: unable to find data in memory cache]" Oct 31 15:25:58.232270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.232253 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:25:58.233704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.233682 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzzr5\" (UniqueName: \"kubernetes.io/projected/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-kube-api-access-jzzr5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.233851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.233833 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.233986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.233950 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:58.468440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.468410 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:58.593093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:58.593059 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:59.105760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.105724 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx" Oct 31 15:25:59.281833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.281809 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:25:59.451326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.451241 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-openshift-ca\") pod \"56877668-57fb-43d1-b731-96593a6c9b6e\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " Oct 31 15:25:59.451556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.451337 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-xks-ca\") pod \"56877668-57fb-43d1-b731-96593a6c9b6e\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " Oct 31 15:25:59.451556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.451508 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56877668-57fb-43d1-b731-96593a6c9b6e-kube-api-access-j8h2x\" (UniqueName: \"kubernetes.io/projected/56877668-57fb-43d1-b731-96593a6c9b6e-kube-api-access-j8h2x\") pod \"56877668-57fb-43d1-b731-96593a6c9b6e\" (UID: \"56877668-57fb-43d1-b731-96593a6c9b6e\") " Oct 31 15:25:59.451717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.451686 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-openshift-ca" pod "56877668-57fb-43d1-b731-96593a6c9b6e" (UID: "56877668-57fb-43d1-b731-96593a6c9b6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:25:59.451884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.451867 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:59.453610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.453589 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-xks-ca" pod "56877668-57fb-43d1-b731-96593a6c9b6e" (UID: "56877668-57fb-43d1-b731-96593a6c9b6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:25:59.453711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.453687 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56877668-57fb-43d1-b731-96593a6c9b6e-kube-api-access-j8h2x" pod "56877668-57fb-43d1-b731-96593a6c9b6e" (UID: "56877668-57fb-43d1-b731-96593a6c9b6e"). InnerVolumeSpecName "kube-api-access-j8h2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:25:59.553576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.553546 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8h2x\" (UniqueName: \"kubernetes.io/projected/56877668-57fb-43d1-b731-96593a6c9b6e-kube-api-access-j8h2x\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:59.553576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.553573 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56877668-57fb-43d1-b731-96593a6c9b6e-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:25:59.704922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.704830 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx"] Oct 31 15:25:59.708906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.708876 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-8l9gx"] Oct 31 15:25:59.817986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.817959 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln"] Oct 31 15:25:59.822401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.822373 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-78fd6998d7-264ln"] Oct 31 15:25:59.859375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.859344 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:25:59.991572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:25:59.991537 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:00.114673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:00.114643 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" Oct 31 15:26:00.114673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:00.114665 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq" event={"ID":"56877668-57fb-43d1-b731-96593a6c9b6e","Type":"ContainerDied","Data":"a174f38edc577ff9f2fc78f80f386b54a9609eb1d3a775996df55ba7eeb65eef"} Oct 31 15:26:00.114870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:00.114708 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a174f38edc577ff9f2fc78f80f386b54a9609eb1d3a775996df55ba7eeb65eef" Oct 31 15:26:00.117124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:00.117104 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:00.496339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:00.496309 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" path="/var/lib/kubelet/pods/aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d/volumes" Oct 31 15:26:00.496762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:00.496747 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd991c02-0fa9-4719-b7c6-0078ba98b9e5" path="/var/lib/kubelet/pods/fd991c02-0fa9-4719-b7c6-0078ba98b9e5/volumes" Oct 31 15:26:03.982786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:03.982751 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs"] Oct 31 15:26:04.093411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:04.093379 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8"] Oct 31 15:26:04.339000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:04.338915 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:04.476223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:04.476192 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:04.617386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:04.617309 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:05.194819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:05.194789 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:08.293399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.293364 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c"] Oct 31 15:26:08.293996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.293978 4865 cpu_manager.go:414] "containerMap: removing container" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" Oct 31 15:26:08.294053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.293998 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" Oct 31 15:26:08.294053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294010 4865 cpu_manager.go:414] "containerMap: removing container" podUID="26c7b7ef-fdb4-48da-9155-0602fcdf0482" containerName="payload-processing" Oct 31 15:26:08.294053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294015 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c7b7ef-fdb4-48da-9155-0602fcdf0482" containerName="payload-processing" Oct 31 15:26:08.294053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294029 4865 cpu_manager.go:414] "containerMap: removing container" podUID="56877668-57fb-43d1-b731-96593a6c9b6e" containerName="revoke-keys" Oct 31 15:26:08.294053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294034 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56877668-57fb-43d1-b731-96593a6c9b6e" containerName="revoke-keys" Oct 31 15:26:08.294218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294060 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fd991c02-0fa9-4719-b7c6-0078ba98b9e5" containerName="payload-pre-processing" Oct 31 15:26:08.294218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294065 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd991c02-0fa9-4719-b7c6-0078ba98b9e5" containerName="payload-pre-processing" Oct 31 15:26:08.294218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294152 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8e00d7-d290-4ef8-86d3-14ebcbf34e4d" containerName="istio-proxy" Oct 31 15:26:08.294218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294163 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c7b7ef-fdb4-48da-9155-0602fcdf0482" containerName="payload-processing" Oct 31 15:26:08.294218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294172 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="56877668-57fb-43d1-b731-96593a6c9b6e" containerName="revoke-keys" Oct 31 15:26:08.294218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.294183 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd991c02-0fa9-4719-b7c6-0078ba98b9e5" containerName="payload-pre-processing" Oct 31 15:26:08.298737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.298719 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.351911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.351880 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:26:08.353814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.353785 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c"] Oct 31 15:26:08.365357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.365325 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.365744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.365504 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r65p\" (UniqueName: \"kubernetes.io/projected/49f54bb4-0364-451b-804e-fb43949a56e8-kube-api-access-2r65p\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.365744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.365588 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.400302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.400271 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:26:08.400302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.400288 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:26:08.400538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.400355 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:26:08.466388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.466353 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.466602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.466503 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.466602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.466564 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r65p\" (UniqueName: \"kubernetes.io/projected/49f54bb4-0364-451b-804e-fb43949a56e8-kube-api-access-2r65p\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.467006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.466984 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.468952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.468930 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.524242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.524206 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r65p\" (UniqueName: \"kubernetes.io/projected/49f54bb4-0364-451b-804e-fb43949a56e8-kube-api-access-2r65p\") pod \"maas-api-revoke-sub-56716f2b-ltl4c\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.612246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.612160 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:08.809197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.809164 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn"] Oct 31 15:26:08.814354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.814336 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.870413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.870331 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn"] Oct 31 15:26:08.871191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.871161 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.871376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.871248 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.871376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.871370 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdtcj\" (UniqueName: \"kubernetes.io/projected/8f9acae8-1773-4465-b14c-d6e9ce158970-kube-api-access-wdtcj\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.972379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.972340 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.972583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.972464 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdtcj\" (UniqueName: \"kubernetes.io/projected/8f9acae8-1773-4465-b14c-d6e9ce158970-kube-api-access-wdtcj\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.972690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.972672 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.973243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.973221 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.974906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.974882 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:08.983849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:08.983817 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" podUID="c20346d1-6975-4d7e-b7f7-240b262139cf" containerName="payload-processing" containerID="cri-o://51a6fff52fef94803fd1c83f3d43137272afa6f46a67e46a92274ed949093566" gracePeriod=25 Oct 31 15:26:09.030427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.030396 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdtcj\" (UniqueName: \"kubernetes.io/projected/8f9acae8-1773-4465-b14c-d6e9ce158970-kube-api-access-wdtcj\") pod \"maas-api-revoke-sub-0a8d4596-lhjrn\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:09.042676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.042652 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c"] Oct 31 15:26:09.061382 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:09.061345 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f54bb4_0364_451b_804e_fb43949a56e8.slice/crio-4c0f76911f6aae80f63d64738c4dd2e89136b068e819af9ce9ffdb3f9f278380 WatchSource:0}: Error finding container 4c0f76911f6aae80f63d64738c4dd2e89136b068e819af9ce9ffdb3f9f278380: Status 404 returned error can't find the container with id 4c0f76911f6aae80f63d64738c4dd2e89136b068e819af9ce9ffdb3f9f278380 Oct 31 15:26:09.094196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.094158 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" podUID="5d646022-1a55-477a-9242-a3db284260bd" containerName="payload-pre-processing" containerID="cri-o://b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770" gracePeriod=25 Oct 31 15:26:09.129501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.129413 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:09.202660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.202551 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" event={"ID":"49f54bb4-0364-451b-804e-fb43949a56e8","Type":"ContainerStarted","Data":"4c0f76911f6aae80f63d64738c4dd2e89136b068e819af9ce9ffdb3f9f278380"} Oct 31 15:26:09.208503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.208245 4865 generic.go:356] "Generic (PLEG): container finished" podID="c20346d1-6975-4d7e-b7f7-240b262139cf" containerID="51a6fff52fef94803fd1c83f3d43137272afa6f46a67e46a92274ed949093566" exitCode=0 Oct 31 15:26:09.208503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.208452 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" event={"ID":"c20346d1-6975-4d7e-b7f7-240b262139cf","Type":"ContainerDied","Data":"51a6fff52fef94803fd1c83f3d43137272afa6f46a67e46a92274ed949093566"} Oct 31 15:26:09.273732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.273700 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:26:09.276769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.276740 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20346d1-6975-4d7e-b7f7-240b262139cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20346d1-6975-4d7e-b7f7-240b262139cf-plugins-config-volume\") pod \"c20346d1-6975-4d7e-b7f7-240b262139cf\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " Oct 31 15:26:09.276970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.276926 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20346d1-6975-4d7e-b7f7-240b262139cf-kube-api-access-8ck5s\" (UniqueName: \"kubernetes.io/projected/c20346d1-6975-4d7e-b7f7-240b262139cf-kube-api-access-8ck5s\") pod \"c20346d1-6975-4d7e-b7f7-240b262139cf\" (UID: \"c20346d1-6975-4d7e-b7f7-240b262139cf\") " Oct 31 15:26:09.277151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.277128 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20346d1-6975-4d7e-b7f7-240b262139cf-plugins-config-volume" pod "c20346d1-6975-4d7e-b7f7-240b262139cf" (UID: "c20346d1-6975-4d7e-b7f7-240b262139cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:09.277537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.277513 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20346d1-6975-4d7e-b7f7-240b262139cf-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:09.279426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.279399 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20346d1-6975-4d7e-b7f7-240b262139cf-kube-api-access-8ck5s" pod "c20346d1-6975-4d7e-b7f7-240b262139cf" (UID: "c20346d1-6975-4d7e-b7f7-240b262139cf"). InnerVolumeSpecName "kube-api-access-8ck5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:09.378872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.378835 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ck5s\" (UniqueName: \"kubernetes.io/projected/c20346d1-6975-4d7e-b7f7-240b262139cf-kube-api-access-8ck5s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:09.396127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.396099 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:26:09.479250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.479216 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d646022-1a55-477a-9242-a3db284260bd-kube-api-access-rb9pk\" (UniqueName: \"kubernetes.io/projected/5d646022-1a55-477a-9242-a3db284260bd-kube-api-access-rb9pk\") pod \"5d646022-1a55-477a-9242-a3db284260bd\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " Oct 31 15:26:09.479250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.479253 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d646022-1a55-477a-9242-a3db284260bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d646022-1a55-477a-9242-a3db284260bd-plugins-config-volume\") pod \"5d646022-1a55-477a-9242-a3db284260bd\" (UID: \"5d646022-1a55-477a-9242-a3db284260bd\") " Oct 31 15:26:09.479721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.479677 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d646022-1a55-477a-9242-a3db284260bd-plugins-config-volume" pod "5d646022-1a55-477a-9242-a3db284260bd" (UID: "5d646022-1a55-477a-9242-a3db284260bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:09.481709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.481548 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d646022-1a55-477a-9242-a3db284260bd-kube-api-access-rb9pk" pod "5d646022-1a55-477a-9242-a3db284260bd" (UID: "5d646022-1a55-477a-9242-a3db284260bd"). InnerVolumeSpecName "kube-api-access-rb9pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:09.522619 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:09.522576 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9acae8_1773_4465_b14c_d6e9ce158970.slice/crio-2edfdec52c8c151cfc3febcb64b97102d44bf763e541173d54a4c5b710ad9e03 WatchSource:0}: Error finding container 2edfdec52c8c151cfc3febcb64b97102d44bf763e541173d54a4c5b710ad9e03: Status 404 returned error can't find the container with id 2edfdec52c8c151cfc3febcb64b97102d44bf763e541173d54a4c5b710ad9e03 Oct 31 15:26:09.524006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.523978 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn"] Oct 31 15:26:09.580716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.580689 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9pk\" (UniqueName: \"kubernetes.io/projected/5d646022-1a55-477a-9242-a3db284260bd-kube-api-access-rb9pk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:09.580814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:09.580716 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d646022-1a55-477a-9242-a3db284260bd-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:10.083972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.083889 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:10.217420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.217389 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" event={"ID":"8f9acae8-1773-4465-b14c-d6e9ce158970","Type":"ContainerStarted","Data":"ab6d807de94a97fdc89a9d6e5d3545984dac08e3384dae81a745325501b46acd"} Oct 31 15:26:10.217588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.217501 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" event={"ID":"8f9acae8-1773-4465-b14c-d6e9ce158970","Type":"ContainerStarted","Data":"2edfdec52c8c151cfc3febcb64b97102d44bf763e541173d54a4c5b710ad9e03"} Oct 31 15:26:10.218954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.218930 4865 generic.go:356] "Generic (PLEG): container finished" podID="49f54bb4-0364-451b-804e-fb43949a56e8" containerID="18f48cde5f3d375a0ab51162979488e8b561880783587ee89fc527a96996cd2f" exitCode=0 Oct 31 15:26:10.219058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.219007 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" event={"ID":"49f54bb4-0364-451b-804e-fb43949a56e8","Type":"ContainerDied","Data":"18f48cde5f3d375a0ab51162979488e8b561880783587ee89fc527a96996cd2f"} Oct 31 15:26:10.220238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.220219 4865 generic.go:356] "Generic (PLEG): container finished" podID="5d646022-1a55-477a-9242-a3db284260bd" containerID="b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770" exitCode=0 Oct 31 15:26:10.220334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.220272 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" Oct 31 15:26:10.220334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.220298 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" event={"ID":"5d646022-1a55-477a-9242-a3db284260bd","Type":"ContainerDied","Data":"b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770"} Oct 31 15:26:10.220501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.220349 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8" event={"ID":"5d646022-1a55-477a-9242-a3db284260bd","Type":"ContainerDied","Data":"c201a18ad4b95e33485c3057b710fedc0ccbc63fea954f4249c92d565a76475c"} Oct 31 15:26:10.220501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.220381 4865 scope.go:122] "RemoveContainer" containerID="b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770" Oct 31 15:26:10.221809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.221789 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" event={"ID":"c20346d1-6975-4d7e-b7f7-240b262139cf","Type":"ContainerDied","Data":"f87a2ca4bcadade0dbb7555e5fdae1c736cde7cb7b3b51c5de58ac452a5911e8"} Oct 31 15:26:10.222027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.222010 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs" Oct 31 15:26:10.248876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.248856 4865 scope.go:122] "RemoveContainer" containerID="b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770" Oct 31 15:26:10.249181 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:26:10.249156 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770\": container with ID starting with b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770 not found: ID does not exist" containerID="b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770" Oct 31 15:26:10.249271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.249194 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770"} err="failed to get container status \"b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770\": rpc error: code = NotFound desc = could not find container \"b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770\": container with ID starting with b59eaec87b829d24a036f56151a47c601a789c14b515f754e2c289026b690770 not found: ID does not exist" Oct 31 15:26:10.249271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.249218 4865 scope.go:122] "RemoveContainer" containerID="51a6fff52fef94803fd1c83f3d43137272afa6f46a67e46a92274ed949093566" Oct 31 15:26:10.336562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.336453 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" podStartSLOduration=2.336439276 podStartE2EDuration="2.336439276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:10.333009639 +0000 UTC m=+2415.952561050" watchObservedRunningTime="2026-10-31 15:26:10.336439276 +0000 UTC m=+2415.955990686" Oct 31 15:26:10.394590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.394564 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:10.565918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.565894 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs"] Oct 31 15:26:10.569448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.569429 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3065a-68d6dffcdf-qzsbs"] Oct 31 15:26:10.679678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.679605 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8"] Oct 31 15:26:10.683286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:10.683265 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3065a-7fb9f87dc5-bsfs8"] Oct 31 15:26:11.231890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.231860 4865 generic.go:356] "Generic (PLEG): container finished" podID="8f9acae8-1773-4465-b14c-d6e9ce158970" containerID="ab6d807de94a97fdc89a9d6e5d3545984dac08e3384dae81a745325501b46acd" exitCode=0 Oct 31 15:26:11.232086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.231949 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" event={"ID":"8f9acae8-1773-4465-b14c-d6e9ce158970","Type":"ContainerDied","Data":"ab6d807de94a97fdc89a9d6e5d3545984dac08e3384dae81a745325501b46acd"} Oct 31 15:26:11.398877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.398850 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:11.504011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.503931 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-xks-ca\") pod \"49f54bb4-0364-451b-804e-fb43949a56e8\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " Oct 31 15:26:11.504184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.504038 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f54bb4-0364-451b-804e-fb43949a56e8-kube-api-access-2r65p\" (UniqueName: \"kubernetes.io/projected/49f54bb4-0364-451b-804e-fb43949a56e8-kube-api-access-2r65p\") pod \"49f54bb4-0364-451b-804e-fb43949a56e8\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " Oct 31 15:26:11.504184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.504115 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-openshift-ca\") pod \"49f54bb4-0364-451b-804e-fb43949a56e8\" (UID: \"49f54bb4-0364-451b-804e-fb43949a56e8\") " Oct 31 15:26:11.504452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.504424 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-openshift-ca" pod "49f54bb4-0364-451b-804e-fb43949a56e8" (UID: "49f54bb4-0364-451b-804e-fb43949a56e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:11.504757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.504738 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:11.506365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.506343 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-xks-ca" pod "49f54bb4-0364-451b-804e-fb43949a56e8" (UID: "49f54bb4-0364-451b-804e-fb43949a56e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:11.506432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.506395 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f54bb4-0364-451b-804e-fb43949a56e8-kube-api-access-2r65p" pod "49f54bb4-0364-451b-804e-fb43949a56e8" (UID: "49f54bb4-0364-451b-804e-fb43949a56e8"). InnerVolumeSpecName "kube-api-access-2r65p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:11.606010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.605975 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f54bb4-0364-451b-804e-fb43949a56e8-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:11.606010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:11.606005 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r65p\" (UniqueName: \"kubernetes.io/projected/49f54bb4-0364-451b-804e-fb43949a56e8-kube-api-access-2r65p\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:12.241110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.241075 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" Oct 31 15:26:12.241294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.241116 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c" event={"ID":"49f54bb4-0364-451b-804e-fb43949a56e8","Type":"ContainerDied","Data":"4c0f76911f6aae80f63d64738c4dd2e89136b068e819af9ce9ffdb3f9f278380"} Oct 31 15:26:12.241294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.241166 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c0f76911f6aae80f63d64738c4dd2e89136b068e819af9ce9ffdb3f9f278380" Oct 31 15:26:12.389668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.389645 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:12.416924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.416893 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9acae8-1773-4465-b14c-d6e9ce158970-kube-api-access-wdtcj\" (UniqueName: \"kubernetes.io/projected/8f9acae8-1773-4465-b14c-d6e9ce158970-kube-api-access-wdtcj\") pod \"8f9acae8-1773-4465-b14c-d6e9ce158970\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " Oct 31 15:26:12.417381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.417011 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-xks-ca\") pod \"8f9acae8-1773-4465-b14c-d6e9ce158970\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " Oct 31 15:26:12.417381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.417137 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-openshift-ca\") pod \"8f9acae8-1773-4465-b14c-d6e9ce158970\" (UID: \"8f9acae8-1773-4465-b14c-d6e9ce158970\") " Oct 31 15:26:12.417517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.417451 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-openshift-ca" pod "8f9acae8-1773-4465-b14c-d6e9ce158970" (UID: "8f9acae8-1773-4465-b14c-d6e9ce158970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:12.417694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.417672 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:12.419227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.419202 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9acae8-1773-4465-b14c-d6e9ce158970-kube-api-access-wdtcj" pod "8f9acae8-1773-4465-b14c-d6e9ce158970" (UID: "8f9acae8-1773-4465-b14c-d6e9ce158970"). InnerVolumeSpecName "kube-api-access-wdtcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:12.419322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.419278 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-xks-ca" pod "8f9acae8-1773-4465-b14c-d6e9ce158970" (UID: "8f9acae8-1773-4465-b14c-d6e9ce158970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:12.496742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.496661 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d646022-1a55-477a-9242-a3db284260bd" path="/var/lib/kubelet/pods/5d646022-1a55-477a-9242-a3db284260bd/volumes" Oct 31 15:26:12.497102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.497088 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20346d1-6975-4d7e-b7f7-240b262139cf" path="/var/lib/kubelet/pods/c20346d1-6975-4d7e-b7f7-240b262139cf/volumes" Oct 31 15:26:12.518732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.518702 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdtcj\" (UniqueName: \"kubernetes.io/projected/8f9acae8-1773-4465-b14c-d6e9ce158970-kube-api-access-wdtcj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:12.518864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:12.518736 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f9acae8-1773-4465-b14c-d6e9ce158970-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:13.249250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:13.249222 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" Oct 31 15:26:13.249433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:13.249216 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn" event={"ID":"8f9acae8-1773-4465-b14c-d6e9ce158970","Type":"ContainerDied","Data":"2edfdec52c8c151cfc3febcb64b97102d44bf763e541173d54a4c5b710ad9e03"} Oct 31 15:26:13.249433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:13.249345 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2edfdec52c8c151cfc3febcb64b97102d44bf763e541173d54a4c5b710ad9e03" Oct 31 15:26:14.150585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.150551 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:14.516624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.516592 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz"] Oct 31 15:26:14.516864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.516823 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" podUID="55409b0d-3c7c-49af-aeb9-fe5ffab8020a" containerName="payload-processing" containerID="cri-o://3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161" gracePeriod=30 Oct 31 15:26:14.568439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.568412 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp"] Oct 31 15:26:14.568653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.568631 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" podUID="f513715b-2cdc-4c65-a0a9-410f44973faa" containerName="payload-pre-processing" containerID="cri-o://bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b" gracePeriod=30 Oct 31 15:26:14.657176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.657143 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:14.783233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.783202 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:14.843803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.843777 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:26:14.872554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.872524 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:26:14.952224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.952197 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f513715b-2cdc-4c65-a0a9-410f44973faa-kube-api-access-mjgrp\" (UniqueName: \"kubernetes.io/projected/f513715b-2cdc-4c65-a0a9-410f44973faa-kube-api-access-mjgrp\") pod \"f513715b-2cdc-4c65-a0a9-410f44973faa\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " Oct 31 15:26:14.952345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.952232 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f513715b-2cdc-4c65-a0a9-410f44973faa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f513715b-2cdc-4c65-a0a9-410f44973faa-plugins-config-volume\") pod \"f513715b-2cdc-4c65-a0a9-410f44973faa\" (UID: \"f513715b-2cdc-4c65-a0a9-410f44973faa\") " Oct 31 15:26:14.952345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.952266 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-plugins-config-volume\") pod \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " Oct 31 15:26:14.952505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.952483 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-kube-api-access-mb7qn\" (UniqueName: \"kubernetes.io/projected/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-kube-api-access-mb7qn\") pod \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\" (UID: \"55409b0d-3c7c-49af-aeb9-fe5ffab8020a\") " Oct 31 15:26:14.952684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.952656 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f513715b-2cdc-4c65-a0a9-410f44973faa-plugins-config-volume" pod "f513715b-2cdc-4c65-a0a9-410f44973faa" (UID: "f513715b-2cdc-4c65-a0a9-410f44973faa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:14.952813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.952749 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-plugins-config-volume" pod "55409b0d-3c7c-49af-aeb9-fe5ffab8020a" (UID: "55409b0d-3c7c-49af-aeb9-fe5ffab8020a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:14.953034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.953017 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f513715b-2cdc-4c65-a0a9-410f44973faa-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:14.953097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.953040 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:14.954505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.954464 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f513715b-2cdc-4c65-a0a9-410f44973faa-kube-api-access-mjgrp" pod "f513715b-2cdc-4c65-a0a9-410f44973faa" (UID: "f513715b-2cdc-4c65-a0a9-410f44973faa"). InnerVolumeSpecName "kube-api-access-mjgrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:14.954606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:14.954587 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-kube-api-access-mb7qn" pod "55409b0d-3c7c-49af-aeb9-fe5ffab8020a" (UID: "55409b0d-3c7c-49af-aeb9-fe5ffab8020a"). InnerVolumeSpecName "kube-api-access-mb7qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:15.063599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.054981 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7qn\" (UniqueName: \"kubernetes.io/projected/55409b0d-3c7c-49af-aeb9-fe5ffab8020a-kube-api-access-mb7qn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:15.063599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.055020 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjgrp\" (UniqueName: \"kubernetes.io/projected/f513715b-2cdc-4c65-a0a9-410f44973faa-kube-api-access-mjgrp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:15.265189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.265150 4865 generic.go:356] "Generic (PLEG): container finished" podID="55409b0d-3c7c-49af-aeb9-fe5ffab8020a" containerID="3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161" exitCode=0 Oct 31 15:26:15.265189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.265192 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" Oct 31 15:26:15.265815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.265244 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" event={"ID":"55409b0d-3c7c-49af-aeb9-fe5ffab8020a","Type":"ContainerDied","Data":"3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161"} Oct 31 15:26:15.265815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.265312 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz" event={"ID":"55409b0d-3c7c-49af-aeb9-fe5ffab8020a","Type":"ContainerDied","Data":"9bded88468fb9cfd045215507249f325dd4be2ea42e424b04995aa72a4a74b93"} Oct 31 15:26:15.265815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.265345 4865 scope.go:122] "RemoveContainer" containerID="3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161" Oct 31 15:26:15.268198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.268172 4865 generic.go:356] "Generic (PLEG): container finished" podID="f513715b-2cdc-4c65-a0a9-410f44973faa" containerID="bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b" exitCode=0 Oct 31 15:26:15.268343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.268227 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" Oct 31 15:26:15.268343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.268268 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" event={"ID":"f513715b-2cdc-4c65-a0a9-410f44973faa","Type":"ContainerDied","Data":"bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b"} Oct 31 15:26:15.268343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.268320 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp" event={"ID":"f513715b-2cdc-4c65-a0a9-410f44973faa","Type":"ContainerDied","Data":"4ebffcf337306549ded23d21e3989bb07a27205edbf77bccf848d51a6c643ec0"} Oct 31 15:26:15.278630 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.278615 4865 scope.go:122] "RemoveContainer" containerID="3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161" Oct 31 15:26:15.278866 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:26:15.278850 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161\": container with ID starting with 3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161 not found: ID does not exist" containerID="3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161" Oct 31 15:26:15.278920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.278876 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161"} err="failed to get container status \"3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161\": rpc error: code = NotFound desc = could not find container \"3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161\": container with ID starting with 3eb201dc8821e6143c8e9513490ef299dc5a5da2368a0db5ba46bee86421b161 not found: ID does not exist" Oct 31 15:26:15.278920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.278891 4865 scope.go:122] "RemoveContainer" containerID="bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b" Oct 31 15:26:15.287825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.287809 4865 scope.go:122] "RemoveContainer" containerID="bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b" Oct 31 15:26:15.288081 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:26:15.288063 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b\": container with ID starting with bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b not found: ID does not exist" containerID="bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b" Oct 31 15:26:15.288136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.288088 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b"} err="failed to get container status \"bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b\": rpc error: code = NotFound desc = could not find container \"bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b\": container with ID starting with bb2a0a8073f1c0681925bfe9c1f60482cf5f773f9f5520ca52e5ecaf5058f82b not found: ID does not exist" Oct 31 15:26:15.434299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.434214 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz"] Oct 31 15:26:15.437834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.437810 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-da9920-654fd58c7b-vpnzz"] Oct 31 15:26:15.548614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.548589 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp"] Oct 31 15:26:15.556409 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:15.555903 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-da9920-7d8f7fd74c-66rnp"] Oct 31 15:26:16.009004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:16.008973 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:16.496831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:16.496800 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55409b0d-3c7c-49af-aeb9-fe5ffab8020a" path="/var/lib/kubelet/pods/55409b0d-3c7c-49af-aeb9-fe5ffab8020a/volumes" Oct 31 15:26:16.497223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:16.497199 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f513715b-2cdc-4c65-a0a9-410f44973faa" path="/var/lib/kubelet/pods/f513715b-2cdc-4c65-a0a9-410f44973faa/volumes" Oct 31 15:26:17.489922 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:26:17.489888 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:26:18.998699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:18.998667 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:26:19.550345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.550309 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg"] Oct 31 15:26:19.551201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551170 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f513715b-2cdc-4c65-a0a9-410f44973faa" containerName="payload-pre-processing" Oct 31 15:26:19.551201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551194 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f513715b-2cdc-4c65-a0a9-410f44973faa" containerName="payload-pre-processing" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551235 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5d646022-1a55-477a-9242-a3db284260bd" containerName="payload-pre-processing" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551245 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d646022-1a55-477a-9242-a3db284260bd" containerName="payload-pre-processing" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551277 4865 cpu_manager.go:414] "containerMap: removing container" podUID="55409b0d-3c7c-49af-aeb9-fe5ffab8020a" containerName="payload-processing" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551286 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55409b0d-3c7c-49af-aeb9-fe5ffab8020a" containerName="payload-processing" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551308 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c20346d1-6975-4d7e-b7f7-240b262139cf" containerName="payload-processing" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551317 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20346d1-6975-4d7e-b7f7-240b262139cf" containerName="payload-processing" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551331 4865 cpu_manager.go:414] "containerMap: removing container" podUID="49f54bb4-0364-451b-804e-fb43949a56e8" containerName="revoke-keys" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551339 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f54bb4-0364-451b-804e-fb43949a56e8" containerName="revoke-keys" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551363 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8f9acae8-1773-4465-b14c-d6e9ce158970" containerName="revoke-keys" Oct 31 15:26:19.551423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551370 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9acae8-1773-4465-b14c-d6e9ce158970" containerName="revoke-keys" Oct 31 15:26:19.552032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551529 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20346d1-6975-4d7e-b7f7-240b262139cf" containerName="payload-processing" Oct 31 15:26:19.552032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551545 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f54bb4-0364-451b-804e-fb43949a56e8" containerName="revoke-keys" Oct 31 15:26:19.552032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551557 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9acae8-1773-4465-b14c-d6e9ce158970" containerName="revoke-keys" Oct 31 15:26:19.552032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551568 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f513715b-2cdc-4c65-a0a9-410f44973faa" containerName="payload-pre-processing" Oct 31 15:26:19.552032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551581 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d646022-1a55-477a-9242-a3db284260bd" containerName="payload-pre-processing" Oct 31 15:26:19.552032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.551593 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="55409b0d-3c7c-49af-aeb9-fe5ffab8020a" containerName="payload-processing" Oct 31 15:26:19.557122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.557096 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.609267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.609237 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg"] Oct 31 15:26:19.615283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.613997 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzzth\" (UniqueName: \"kubernetes.io/projected/3887ea21-4360-4fb9-93dc-d634b6b29b2a-kube-api-access-gzzth\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.615283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.614267 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.615283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.614353 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.653965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.653546 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:26:19.653965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.654068 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:26:19.654398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.654161 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:26:19.654398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.654181 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:26:19.716004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.715938 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.716217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.716087 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzzth\" (UniqueName: \"kubernetes.io/projected/3887ea21-4360-4fb9-93dc-d634b6b29b2a-kube-api-access-gzzth\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.716347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.716323 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.717530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.717504 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.722309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.722282 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.774759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.774722 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzzth\" (UniqueName: \"kubernetes.io/projected/3887ea21-4360-4fb9-93dc-d634b6b29b2a-kube-api-access-gzzth\") pod \"maas-api-revoke-sub-0cc63d17-rkfzg\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:19.891549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:19.880710 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:20.462490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:20.462025 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg"] Oct 31 15:26:21.089253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.088238 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:21.189494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.188565 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:21.214544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.198152 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:26:21.534494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.533921 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" event={"ID":"3887ea21-4360-4fb9-93dc-d634b6b29b2a","Type":"ContainerStarted","Data":"4c5180a94689f5eb01fe378aaafcd83c64c067de4b04ce3e5e77618934aaa07c"} Oct 31 15:26:21.534494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.533998 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" event={"ID":"3887ea21-4360-4fb9-93dc-d634b6b29b2a","Type":"ContainerStarted","Data":"3eb804f3462ea9d6c1ed9138382655f0996ec461f488e16e297fa3850e46fd9e"} Oct 31 15:26:21.655497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.654547 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:26:21.720496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.706048 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:26:21.720496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.706110 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:21.720496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.706408 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:26:21.720496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.706442 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:21.917656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.912945 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:26:21.917656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:21.913786 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:26:22.021495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.020976 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:26:22.031508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.029553 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.031508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.030280 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:26:22.097927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.089553 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.097927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.089925 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:26:22.183878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.183724 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.199493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.189548 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:26:22.199493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.189979 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:26:22.200423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.200398 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:26:22.224522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.200937 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:26:22.334790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.334731 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" podStartSLOduration=3.334682011 podStartE2EDuration="3.334682011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:21.655708174 +0000 UTC m=+2427.275259586" watchObservedRunningTime="2026-10-31 15:26:22.334682011 +0000 UTC m=+2427.954233423" Oct 31 15:26:22.713495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.657378 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.914638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.913677 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.914638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:22.914197 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:23.048125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.048021 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:26:23.048767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.048737 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:26:23.067959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.067924 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:26:23.068250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.068231 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:26:23.068414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.068400 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:26:23.068592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.068577 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:26:23.068734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.068722 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:26:23.072448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.072417 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:23.072918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.072896 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:26:23.077076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.077052 4865 generic.go:356] "Generic (PLEG): container finished" podID="3887ea21-4360-4fb9-93dc-d634b6b29b2a" containerID="4c5180a94689f5eb01fe378aaafcd83c64c067de4b04ce3e5e77618934aaa07c" exitCode=0 Oct 31 15:26:23.077585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.077559 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" event={"ID":"3887ea21-4360-4fb9-93dc-d634b6b29b2a","Type":"ContainerDied","Data":"4c5180a94689f5eb01fe378aaafcd83c64c067de4b04ce3e5e77618934aaa07c"} Oct 31 15:26:23.095592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.095550 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:23.096160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.096129 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:26:23.100496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.096637 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:26:23.100705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.100688 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:26:23.196428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.193611 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:23.204585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.204547 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:23.275757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.275393 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:26:23.320784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.316388 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:23.368260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.368232 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:26:23.421508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.414340 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:26:23.436268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.434971 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:26:23.444091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.444066 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:26:23.447947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.447872 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:23.487188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.487159 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:26:23.515072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.515044 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:26:23.526410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.523903 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:26:23.536792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.536766 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:26:23.565044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:23.565012 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:26:24.242570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.242545 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:24.322394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.322362 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3887ea21-4360-4fb9-93dc-d634b6b29b2a-kube-api-access-gzzth\" (UniqueName: \"kubernetes.io/projected/3887ea21-4360-4fb9-93dc-d634b6b29b2a-kube-api-access-gzzth\") pod \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " Oct 31 15:26:24.322610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.322424 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-openshift-ca\") pod \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " Oct 31 15:26:24.322610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.322511 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-xks-ca\") pod \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\" (UID: \"3887ea21-4360-4fb9-93dc-d634b6b29b2a\") " Oct 31 15:26:24.322841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.322814 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-openshift-ca" pod "3887ea21-4360-4fb9-93dc-d634b6b29b2a" (UID: "3887ea21-4360-4fb9-93dc-d634b6b29b2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:24.324718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.324697 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-xks-ca" pod "3887ea21-4360-4fb9-93dc-d634b6b29b2a" (UID: "3887ea21-4360-4fb9-93dc-d634b6b29b2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:24.324814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.324780 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3887ea21-4360-4fb9-93dc-d634b6b29b2a-kube-api-access-gzzth" pod "3887ea21-4360-4fb9-93dc-d634b6b29b2a" (UID: "3887ea21-4360-4fb9-93dc-d634b6b29b2a"). InnerVolumeSpecName "kube-api-access-gzzth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:24.424321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.424238 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzzth\" (UniqueName: \"kubernetes.io/projected/3887ea21-4360-4fb9-93dc-d634b6b29b2a-kube-api-access-gzzth\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:24.424321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.424264 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:24.424321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.424273 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3887ea21-4360-4fb9-93dc-d634b6b29b2a-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:24.883030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:24.882998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:25.094261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:25.094225 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" Oct 31 15:26:25.094461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:25.094261 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg" event={"ID":"3887ea21-4360-4fb9-93dc-d634b6b29b2a","Type":"ContainerDied","Data":"3eb804f3462ea9d6c1ed9138382655f0996ec461f488e16e297fa3850e46fd9e"} Oct 31 15:26:25.094461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:25.094305 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb804f3462ea9d6c1ed9138382655f0996ec461f488e16e297fa3850e46fd9e" Oct 31 15:26:25.452850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:25.452820 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:25.685019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:25.684986 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:26.204446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:26.204414 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:27.642898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:27.642859 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:29.650849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.650820 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs"] Oct 31 15:26:29.651389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.651376 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3887ea21-4360-4fb9-93dc-d634b6b29b2a" containerName="revoke-keys" Oct 31 15:26:29.651431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.651390 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3887ea21-4360-4fb9-93dc-d634b6b29b2a" containerName="revoke-keys" Oct 31 15:26:29.651551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.651539 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3887ea21-4360-4fb9-93dc-d634b6b29b2a" containerName="revoke-keys" Oct 31 15:26:29.655433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.655418 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.699980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.699941 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.700161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.699998 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zbn4\" (UniqueName: \"kubernetes.io/projected/593ce39c-8aff-4e5b-9084-21dec0f18089-kube-api-access-2zbn4\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.700161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.700116 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.711456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.711419 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs"] Oct 31 15:26:29.756663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.756631 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:26:29.756663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.756633 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:26:29.756663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.756667 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:26:29.756923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.756674 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:26:29.801556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.801521 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.801750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.801566 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zbn4\" (UniqueName: \"kubernetes.io/projected/593ce39c-8aff-4e5b-9084-21dec0f18089-kube-api-access-2zbn4\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.801750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.801703 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.802250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.802220 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.804240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.804217 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.859332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.859301 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zbn4\" (UniqueName: \"kubernetes.io/projected/593ce39c-8aff-4e5b-9084-21dec0f18089-kube-api-access-2zbn4\") pod \"maas-api-revoke-sub-6e4aa939-62cxs\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:29.968206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:29.968175 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:30.342352 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:30.342323 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593ce39c_8aff_4e5b_9084_21dec0f18089.slice/crio-757d299f7e032e3f9319a66534280a76880f903c60683f852de811833ae3c4b2 WatchSource:0}: Error finding container 757d299f7e032e3f9319a66534280a76880f903c60683f852de811833ae3c4b2: Status 404 returned error can't find the container with id 757d299f7e032e3f9319a66534280a76880f903c60683f852de811833ae3c4b2 Oct 31 15:26:30.344048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:30.344023 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs"] Oct 31 15:26:31.140068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:31.140029 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" event={"ID":"593ce39c-8aff-4e5b-9084-21dec0f18089","Type":"ContainerStarted","Data":"3463c6308d9c09c94bb8e52005a1fcb9971d61f5927021a5b932f520ad52e24f"} Oct 31 15:26:31.140572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:31.140094 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" event={"ID":"593ce39c-8aff-4e5b-9084-21dec0f18089","Type":"ContainerStarted","Data":"757d299f7e032e3f9319a66534280a76880f903c60683f852de811833ae3c4b2"} Oct 31 15:26:31.256234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:31.256185 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" podStartSLOduration=2.256171658 podStartE2EDuration="2.256171658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:31.254622494 +0000 UTC m=+2436.874173907" watchObservedRunningTime="2026-10-31 15:26:31.256171658 +0000 UTC m=+2436.875723068" Oct 31 15:26:31.822101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:31.822068 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:32.149269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:32.149192 4865 generic.go:356] "Generic (PLEG): container finished" podID="593ce39c-8aff-4e5b-9084-21dec0f18089" containerID="3463c6308d9c09c94bb8e52005a1fcb9971d61f5927021a5b932f520ad52e24f" exitCode=0 Oct 31 15:26:32.149269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:32.149237 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" event={"ID":"593ce39c-8aff-4e5b-9084-21dec0f18089","Type":"ContainerDied","Data":"3463c6308d9c09c94bb8e52005a1fcb9971d61f5927021a5b932f520ad52e24f"} Oct 31 15:26:32.663208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:32.663176 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:33.305090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.305065 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:33.457806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.457779 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593ce39c-8aff-4e5b-9084-21dec0f18089-kube-api-access-2zbn4\" (UniqueName: \"kubernetes.io/projected/593ce39c-8aff-4e5b-9084-21dec0f18089-kube-api-access-2zbn4\") pod \"593ce39c-8aff-4e5b-9084-21dec0f18089\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " Oct 31 15:26:33.457977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.457838 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-openshift-ca\") pod \"593ce39c-8aff-4e5b-9084-21dec0f18089\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " Oct 31 15:26:33.457977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.457879 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-xks-ca\") pod \"593ce39c-8aff-4e5b-9084-21dec0f18089\" (UID: \"593ce39c-8aff-4e5b-9084-21dec0f18089\") " Oct 31 15:26:33.458325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.458296 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-openshift-ca" pod "593ce39c-8aff-4e5b-9084-21dec0f18089" (UID: "593ce39c-8aff-4e5b-9084-21dec0f18089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:33.460223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.460197 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593ce39c-8aff-4e5b-9084-21dec0f18089-kube-api-access-2zbn4" pod "593ce39c-8aff-4e5b-9084-21dec0f18089" (UID: "593ce39c-8aff-4e5b-9084-21dec0f18089"). InnerVolumeSpecName "kube-api-access-2zbn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:33.460653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.460631 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-xks-ca" pod "593ce39c-8aff-4e5b-9084-21dec0f18089" (UID: "593ce39c-8aff-4e5b-9084-21dec0f18089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:33.559570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.559540 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zbn4\" (UniqueName: \"kubernetes.io/projected/593ce39c-8aff-4e5b-9084-21dec0f18089-kube-api-access-2zbn4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:33.559570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.559564 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:33.559570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.559573 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/593ce39c-8aff-4e5b-9084-21dec0f18089-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:33.835318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.835229 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:33.985013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:33.984984 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:34.164209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:34.164122 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" Oct 31 15:26:34.164209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:34.164156 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs" event={"ID":"593ce39c-8aff-4e5b-9084-21dec0f18089","Type":"ContainerDied","Data":"757d299f7e032e3f9319a66534280a76880f903c60683f852de811833ae3c4b2"} Oct 31 15:26:34.164451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:34.164231 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="757d299f7e032e3f9319a66534280a76880f903c60683f852de811833ae3c4b2" Oct 31 15:26:35.438093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:35.438059 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:35.906760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:35.906725 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:35.916333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:35.916310 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55"] Oct 31 15:26:35.916548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:35.916520 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" containerID="cri-o://3071589ece45f2eac0abec6f53695e0c92a5df07414c1b97d25321b2d015fd0f" gracePeriod=30 Oct 31 15:26:36.622813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:36.622780 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:36.745162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:36.745129 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:36.951076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:36.950984 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:37.228064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.228028 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:37.428017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.427979 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz"] Oct 31 15:26:37.428618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.428601 4865 cpu_manager.go:414] "containerMap: removing container" podUID="593ce39c-8aff-4e5b-9084-21dec0f18089" containerName="revoke-keys" Oct 31 15:26:37.428672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.428619 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593ce39c-8aff-4e5b-9084-21dec0f18089" containerName="revoke-keys" Oct 31 15:26:37.428727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.428718 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="593ce39c-8aff-4e5b-9084-21dec0f18089" containerName="revoke-keys" Oct 31 15:26:37.432340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.432323 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.485857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.485792 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-531c3c-openshift-default-dockercfg-x6fjr\"" Oct 31 15:26:37.514383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.514356 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:37.532216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532190 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-istio-token\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532229 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924114e3-1c2c-432e-8492-b7f6811359ea-istiod-ca-cert\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532264 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-envoy\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532368 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-credential-socket\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532392 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-socket\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532411 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924114e3-1c2c-432e-8492-b7f6811359ea-istio-podinfo\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532561 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st22\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-kube-api-access-5st22\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532658 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-certs\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.532752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.532700 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-data\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.542077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.542049 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz"] Oct 31 15:26:37.638633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.638595 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-socket\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.660982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.660945 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924114e3-1c2c-432e-8492-b7f6811359ea-istio-podinfo\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.661810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.653331 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-socket\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.662110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.662090 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st22\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-kube-api-access-5st22\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.662709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.662687 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-certs\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.662946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.662930 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-data\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.663222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.663200 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-istio-token\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.663386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.663370 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924114e3-1c2c-432e-8492-b7f6811359ea-istiod-ca-cert\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.663531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.663516 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-envoy\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.666108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.663630 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-data\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.666108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.663672 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-credential-socket\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.666108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.663015 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-certs\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.666108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.664092 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-credential-socket\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.667277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.667253 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924114e3-1c2c-432e-8492-b7f6811359ea-istiod-ca-cert\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.685167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.682851 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-envoy\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.692806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.692721 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924114e3-1c2c-432e-8492-b7f6811359ea-istio-podinfo\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.720940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.720910 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st22\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-kube-api-access-5st22\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.722178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.722159 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-istio-token\") pod \"e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.746828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.746743 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:37.812496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:37.812268 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:38.253267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.253235 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz"] Oct 31 15:26:38.291189 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:38.291162 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924114e3_1c2c_432e_8492_b7f6811359ea.slice/crio-9ceb77ab324e5f5ba9f3281eb8c83fea4f67347bdbb817be0580139505494769 WatchSource:0}: Error finding container 9ceb77ab324e5f5ba9f3281eb8c83fea4f67347bdbb817be0580139505494769: Status 404 returned error can't find the container with id 9ceb77ab324e5f5ba9f3281eb8c83fea4f67347bdbb817be0580139505494769 Oct 31 15:26:38.293941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.293908 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:38.294034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.293976 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:38.294034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.294006 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:38.309174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.309152 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf"] Oct 31 15:26:38.314649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.314632 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:38.367668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.367640 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c4ede8\"" Oct 31 15:26:38.369086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.368182 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c4ede8-dockercfg-bgs4g\"" Oct 31 15:26:38.370627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.370606 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf"] Oct 31 15:26:38.401615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.401573 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eac2677-04ee-4ab1-85aa-925e2a239550-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:38.401975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.401947 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq48m\" (UniqueName: \"kubernetes.io/projected/4eac2677-04ee-4ab1-85aa-925e2a239550-kube-api-access-gq48m\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:38.503771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.503691 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eac2677-04ee-4ab1-85aa-925e2a239550-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:38.503771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.503757 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq48m\" (UniqueName: \"kubernetes.io/projected/4eac2677-04ee-4ab1-85aa-925e2a239550-kube-api-access-gq48m\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:38.504435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.504409 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eac2677-04ee-4ab1-85aa-925e2a239550-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:38.561930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.561896 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq48m\" (UniqueName: \"kubernetes.io/projected/4eac2677-04ee-4ab1-85aa-925e2a239550-kube-api-access-gq48m\") pod \"payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:38.648140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:38.648105 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:39.027971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.027937 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t"] Oct 31 15:26:39.033432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.033410 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.034507 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:39.034479 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eac2677_04ee_4ab1_85aa_925e2a239550.slice/crio-762fb6aa74f403239776e7f7f3953ebc5976739c3b0ba4b7b0aa3df26a505566 WatchSource:0}: Error finding container 762fb6aa74f403239776e7f7f3953ebc5976739c3b0ba4b7b0aa3df26a505566: Status 404 returned error can't find the container with id 762fb6aa74f403239776e7f7f3953ebc5976739c3b0ba4b7b0aa3df26a505566 Oct 31 15:26:39.034710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.034691 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf"] Oct 31 15:26:39.089623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.089593 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t"] Oct 31 15:26:39.110769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.110739 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9g2g\" (UniqueName: \"kubernetes.io/projected/0b141460-e88a-45b2-b276-140584ae230a-kube-api-access-g9g2g\") pod \"payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.110903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.110781 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b141460-e88a-45b2-b276-140584ae230a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.202385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.202346 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" event={"ID":"924114e3-1c2c-432e-8492-b7f6811359ea","Type":"ContainerStarted","Data":"b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605"} Oct 31 15:26:39.202573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.202410 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" event={"ID":"924114e3-1c2c-432e-8492-b7f6811359ea","Type":"ContainerStarted","Data":"9ceb77ab324e5f5ba9f3281eb8c83fea4f67347bdbb817be0580139505494769"} Oct 31 15:26:39.204185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.204155 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" event={"ID":"4eac2677-04ee-4ab1-85aa-925e2a239550","Type":"ContainerStarted","Data":"c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05"} Oct 31 15:26:39.204319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.204207 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" event={"ID":"4eac2677-04ee-4ab1-85aa-925e2a239550","Type":"ContainerStarted","Data":"762fb6aa74f403239776e7f7f3953ebc5976739c3b0ba4b7b0aa3df26a505566"} Oct 31 15:26:39.204319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.204230 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:39.212223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.212200 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9g2g\" (UniqueName: \"kubernetes.io/projected/0b141460-e88a-45b2-b276-140584ae230a-kube-api-access-g9g2g\") pod \"payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.212323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.212237 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b141460-e88a-45b2-b276-140584ae230a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.212986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.212963 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b141460-e88a-45b2-b276-140584ae230a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.269847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.269823 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9g2g\" (UniqueName: \"kubernetes.io/projected/0b141460-e88a-45b2-b276-140584ae230a-kube-api-access-g9g2g\") pod \"payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.326287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.325938 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podStartSLOduration=2.325921046 podStartE2EDuration="2.325921046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:39.321010246 +0000 UTC m=+2444.940561659" watchObservedRunningTime="2026-10-31 15:26:39.325921046 +0000 UTC m=+2444.945472459" Oct 31 15:26:39.369482 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.369438 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:39.455498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.445513 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" podStartSLOduration=1.445493321 podStartE2EDuration="1.445493321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:39.441252263 +0000 UTC m=+2445.060803676" watchObservedRunningTime="2026-10-31 15:26:39.445493321 +0000 UTC m=+2445.065044729" Oct 31 15:26:39.747185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.747150 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:39.751291 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:39.751259 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b141460_e88a_45b2_b276_140584ae230a.slice/crio-5f9e90de41ecda55ece3ec819b129af9e6ac7cc5f3cdf35af69755d68ed70066 WatchSource:0}: Error finding container 5f9e90de41ecda55ece3ec819b129af9e6ac7cc5f3cdf35af69755d68ed70066: Status 404 returned error can't find the container with id 5f9e90de41ecda55ece3ec819b129af9e6ac7cc5f3cdf35af69755d68ed70066 Oct 31 15:26:39.753172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.753146 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t"] Oct 31 15:26:39.753294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.753274 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:39.837926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.837900 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd"] Oct 31 15:26:39.841896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.841877 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:39.897782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.897751 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd"] Oct 31 15:26:39.900314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.900267 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:39.910836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.910811 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:26:39.911034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.910892 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:26:39.911116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.910892 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:26:39.913862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.913810 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:26:39.923890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.923858 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqcrk\" (UniqueName: \"kubernetes.io/projected/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-kube-api-access-cqcrk\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:39.924175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.924156 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:39.924359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:39.924344 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.025441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.025356 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqcrk\" (UniqueName: \"kubernetes.io/projected/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-kube-api-access-cqcrk\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.025786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.025524 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.025953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.025926 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.026357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.026329 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.028756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.028730 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.082982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.082952 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqcrk\" (UniqueName: \"kubernetes.io/projected/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-kube-api-access-cqcrk\") pod \"maas-api-revoke-sub-7e9a0904-2fhmd\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.158175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.158145 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:40.238585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.237956 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" event={"ID":"0b141460-e88a-45b2-b276-140584ae230a","Type":"ContainerStarted","Data":"aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85"} Oct 31 15:26:40.238585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.238069 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" event={"ID":"0b141460-e88a-45b2-b276-140584ae230a","Type":"ContainerStarted","Data":"5f9e90de41ecda55ece3ec819b129af9e6ac7cc5f3cdf35af69755d68ed70066"} Oct 31 15:26:40.238585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.238121 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:40.239387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.239210 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:40.240726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.240335 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:26:40.358797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.358705 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" podStartSLOduration=2.358692481 podStartE2EDuration="2.358692481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:40.354350004 +0000 UTC m=+2445.973901419" watchObservedRunningTime="2026-10-31 15:26:40.358692481 +0000 UTC m=+2445.978243889" Oct 31 15:26:40.660640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:40.660608 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd"] Oct 31 15:26:41.321196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:41.320841 4865 generic.go:356] "Generic (PLEG): container finished" podID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerID="3071589ece45f2eac0abec6f53695e0c92a5df07414c1b97d25321b2d015fd0f" exitCode=0 Oct 31 15:26:41.321196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:41.321010 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" event={"ID":"69f26fd5-25d9-41c0-bd46-b6d03361bfe1","Type":"ContainerDied","Data":"3071589ece45f2eac0abec6f53695e0c92a5df07414c1b97d25321b2d015fd0f"} Oct 31 15:26:41.321196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:41.321075 4865 scope.go:122] "RemoveContainer" containerID="173c81ec60901bb2bcd39d25a2012947e37757f69a8cf6b5396f139a4a3b4734" Oct 31 15:26:41.356073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:41.356019 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" event={"ID":"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4","Type":"ContainerStarted","Data":"39e70dfa09f3bb402212c2fc52687268d0d0a79b8382111f4910b98af5916c46"} Oct 31 15:26:41.404927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:41.404847 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:41.561564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:41.560683 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:42.273510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.258374 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:42.298523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.290546 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:42.383519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.382654 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:42.444151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.444012 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 15:26:42.469925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.468560 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:42.469925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.468817 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:26:42.498509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.491651 4865 generic.go:356] "Generic (PLEG): container finished" podID="6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" containerID="c1c442b857cbbf656ca384978823ad1071743298075a01c61d53f5e51446a256" exitCode=0 Oct 31 15:26:42.513059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.498829 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:26:42.513059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.504997 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" event={"ID":"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4","Type":"ContainerDied","Data":"c1c442b857cbbf656ca384978823ad1071743298075a01c61d53f5e51446a256"} Oct 31 15:26:42.535329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.530955 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:42.565606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.565556 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:42.568662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.565664 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:42.568662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.568560 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:26:42.579502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.570346 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:42.691494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.688322 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istiod-ca-cert\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.726976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.726944 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-certs\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.727249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.727224 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-socket\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.727440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.727421 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-data\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.727620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.727602 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-podinfo\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.728749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.728730 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-token\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-token\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.728950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.728935 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-envoy\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.729055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.729043 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-kube-api-access-46599\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-kube-api-access-46599\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.729196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.729184 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-credential-socket\") pod \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\" (UID: \"69f26fd5-25d9-41c0-bd46-b6d03361bfe1\") " Oct 31 15:26:42.752496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.726849 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istiod-ca-cert" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:42.781580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.728011 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-certs" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:26:42.781829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.728189 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-socket" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:26:42.781975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.728363 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-data" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:26:42.782071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.747531 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-credential-socket" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:26:42.782175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.760781 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-podinfo" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:26:42.825724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.821695 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-token" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:42.825724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.825638 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-kube-api-access-46599" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "kube-api-access-46599". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:42.825724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.825637 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-envoy" pod "69f26fd5-25d9-41c0-bd46-b6d03361bfe1" (UID: "69f26fd5-25d9-41c0-bd46-b6d03361bfe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842611 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842642 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842660 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842674 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842687 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842700 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842716 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842730 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46599\" (UniqueName: \"kubernetes.io/projected/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-kube-api-access-46599\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.845493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.842746 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f26fd5-25d9-41c0-bd46-b6d03361bfe1-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:42.886306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:42.886268 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:43.259609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.259565 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.304649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.304598 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.306009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.305986 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:26:43.386777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.386737 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:26:43.388057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.388026 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:26:43.480496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.447548 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.480496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.448115 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:26:43.511551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.483022 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.511980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.511941 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.512741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.512717 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:26:43.512929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.512913 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:26:43.566520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.549222 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 15:26:43.627503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.587833 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.627503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.587872 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:26:43.627503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.610448 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:26:43.627503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.610536 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:26:43.627503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.589332 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 15:26:43.711535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.711488 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 15:26:43.750024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.749991 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" Oct 31 15:26:43.850229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.850195 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:26:43.850537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.850508 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55" event={"ID":"69f26fd5-25d9-41c0-bd46-b6d03361bfe1","Type":"ContainerDied","Data":"9ca2b9df95ea044bf7490ae783a790812dd1dccaf2cff384280471ddf9623ad2"} Oct 31 15:26:43.854592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.854565 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:26:43.872495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:43.856045 4865 scope.go:122] "RemoveContainer" containerID="3071589ece45f2eac0abec6f53695e0c92a5df07414c1b97d25321b2d015fd0f" Oct 31 15:26:44.148343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.147015 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:26:44.265191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.265148 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:26:44.267674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.267644 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:26:44.295500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.284320 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:44.314495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.310003 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 15:26:44.404690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.397574 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:44.619497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.618950 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:44.667879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.663912 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:44.748769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.744614 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 15:26:44.748769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.747460 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:26:44.863014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:44.862809 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:45.013547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.008935 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:45.126533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.123691 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:45.233788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.233745 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:45.295184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.295091 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:26:45.406731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.406680 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" podUID="a447a957-8083-43b4-974e-0cc8f358cdba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded" Oct 31 15:26:45.533516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.518716 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:45.533516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.518870 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:45.533516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.527798 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:45.533516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.530411 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:45.673187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.673094 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 15:26:45.673554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.673533 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 15:26:45.673701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.673686 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 15:26:45.673804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.673790 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 15:26:45.673893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.673829 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb" Oct 31 15:26:45.673893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.673863 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:26:45.691485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.689376 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 15:26:45.691764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.691749 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:45.699349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.695268 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:26:45.702095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.702079 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:26:45.848827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.848798 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-openshift-ca\") pod \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " Oct 31 15:26:45.849011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.848852 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-kube-api-access-cqcrk\" (UniqueName: \"kubernetes.io/projected/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-kube-api-access-cqcrk\") pod \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " Oct 31 15:26:45.849057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.849013 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-xks-ca\") pod \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\" (UID: \"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4\") " Oct 31 15:26:45.849188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.849151 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-openshift-ca" pod "6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" (UID: "6890c1a1-c2cf-4eed-a25f-6807c65fb9c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:45.850603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.850580 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:45.851452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.851425 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-xks-ca" pod "6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" (UID: "6890c1a1-c2cf-4eed-a25f-6807c65fb9c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:45.851582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.851564 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-kube-api-access-cqcrk" pod "6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" (UID: "6890c1a1-c2cf-4eed-a25f-6807c65fb9c4"). InnerVolumeSpecName "kube-api-access-cqcrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:45.873377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.873347 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55"] Oct 31 15:26:45.922281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.922250 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3065a-openshift-default-898cf5df7-k5n55"] Oct 31 15:26:45.951958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.951886 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:45.951958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:45.951915 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqcrk\" (UniqueName: \"kubernetes.io/projected/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4-kube-api-access-cqcrk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:46.217151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:46.217110 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:46.497496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:46.497409 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" path="/var/lib/kubelet/pods/69f26fd5-25d9-41c0-bd46-b6d03361bfe1/volumes" Oct 31 15:26:46.702755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:46.702724 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" Oct 31 15:26:46.702931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:46.702773 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd" event={"ID":"6890c1a1-c2cf-4eed-a25f-6807c65fb9c4","Type":"ContainerDied","Data":"39e70dfa09f3bb402212c2fc52687268d0d0a79b8382111f4910b98af5916c46"} Oct 31 15:26:46.702931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:46.702835 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e70dfa09f3bb402212c2fc52687268d0d0a79b8382111f4910b98af5916c46" Oct 31 15:26:47.118958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:47.118927 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:47.490513 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:26:47.490451 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:26:47.881317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:47.881216 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:48.002076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.002039 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:48.408828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.408796 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn"] Oct 31 15:26:48.413636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.409559 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" containerName="revoke-keys" Oct 31 15:26:48.413636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.409587 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" containerName="revoke-keys" Oct 31 15:26:48.413636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.409615 4865 cpu_manager.go:414] "containerMap: removing container" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" Oct 31 15:26:48.413636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.409622 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" Oct 31 15:26:48.413636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.409883 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" Oct 31 15:26:48.413636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.409895 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" containerName="revoke-keys" Oct 31 15:26:48.413636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.409908 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" Oct 31 15:26:48.443892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.443860 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.455657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.455609 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn"] Oct 31 15:26:48.499277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.499240 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-531c3c-dockercfg-s76gw\"" Oct 31 15:26:48.499503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.499314 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-531c3c\"" Oct 31 15:26:48.532892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.532861 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx"] Oct 31 15:26:48.533758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.533739 4865 cpu_manager.go:414] "containerMap: removing container" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" Oct 31 15:26:48.533839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.533759 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f26fd5-25d9-41c0-bd46-b6d03361bfe1" containerName="istio-proxy" Oct 31 15:26:48.537530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.537510 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:48.591829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.591794 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx"] Oct 31 15:26:48.599646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.599621 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxl2\" (UniqueName: \"kubernetes.io/projected/c7049ebd-d545-4708-aec0-77058dd16d15-kube-api-access-vkxl2\") pod \"payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.599816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.599799 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7049ebd-d545-4708-aec0-77058dd16d15-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.701132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.701038 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c23a96-8377-428b-942a-8bada9964aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:48.701315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.701224 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxl2\" (UniqueName: \"kubernetes.io/projected/c7049ebd-d545-4708-aec0-77058dd16d15-kube-api-access-vkxl2\") pod \"payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.701396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.701325 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqntc\" (UniqueName: \"kubernetes.io/projected/f9c23a96-8377-428b-942a-8bada9964aba-kube-api-access-sqntc\") pod \"payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:48.701501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.701484 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7049ebd-d545-4708-aec0-77058dd16d15-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.702087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.702068 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7049ebd-d545-4708-aec0-77058dd16d15-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.759287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.759261 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxl2\" (UniqueName: \"kubernetes.io/projected/c7049ebd-d545-4708-aec0-77058dd16d15-kube-api-access-vkxl2\") pod \"payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.771170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.771145 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:48.802939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.802908 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqntc\" (UniqueName: \"kubernetes.io/projected/f9c23a96-8377-428b-942a-8bada9964aba-kube-api-access-sqntc\") pod \"payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:48.803088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.803019 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c23a96-8377-428b-942a-8bada9964aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:48.803682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.803658 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c23a96-8377-428b-942a-8bada9964aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:48.848969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.848944 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:48.860523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:48.860499 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqntc\" (UniqueName: \"kubernetes.io/projected/f9c23a96-8377-428b-942a-8bada9964aba-kube-api-access-sqntc\") pod \"payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:49.147660 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:49.147630 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7049ebd_d545_4708_aec0_77058dd16d15.slice/crio-f78579e3c0efe4212b054a5cd78906a0bb0ca97eddced0b308eaa5dc9f1b3bad WatchSource:0}: Error finding container f78579e3c0efe4212b054a5cd78906a0bb0ca97eddced0b308eaa5dc9f1b3bad: Status 404 returned error can't find the container with id f78579e3c0efe4212b054a5cd78906a0bb0ca97eddced0b308eaa5dc9f1b3bad Oct 31 15:26:49.148309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.148284 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn"] Oct 31 15:26:49.151149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.151015 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:49.539554 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:49.539522 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c23a96_8377_428b_942a_8bada9964aba.slice/crio-104a6f5f6eb6732a8b8a8cf37bca16683664235e511162e311b35eb93cdaf763 WatchSource:0}: Error finding container 104a6f5f6eb6732a8b8a8cf37bca16683664235e511162e311b35eb93cdaf763: Status 404 returned error can't find the container with id 104a6f5f6eb6732a8b8a8cf37bca16683664235e511162e311b35eb93cdaf763 Oct 31 15:26:49.541322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.541290 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx"] Oct 31 15:26:49.731269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.731232 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" event={"ID":"c7049ebd-d545-4708-aec0-77058dd16d15","Type":"ContainerStarted","Data":"5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d"} Oct 31 15:26:49.731443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.731324 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" event={"ID":"c7049ebd-d545-4708-aec0-77058dd16d15","Type":"ContainerStarted","Data":"f78579e3c0efe4212b054a5cd78906a0bb0ca97eddced0b308eaa5dc9f1b3bad"} Oct 31 15:26:49.731564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.731443 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:26:49.732968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.732942 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" event={"ID":"f9c23a96-8377-428b-942a-8bada9964aba","Type":"ContainerStarted","Data":"fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0"} Oct 31 15:26:49.733115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.732979 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" event={"ID":"f9c23a96-8377-428b-942a-8bada9964aba","Type":"ContainerStarted","Data":"104a6f5f6eb6732a8b8a8cf37bca16683664235e511162e311b35eb93cdaf763"} Oct 31 15:26:49.733115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.733086 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:26:49.849407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.849312 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" podStartSLOduration=1.849296588 podStartE2EDuration="1.849296588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:49.845728214 +0000 UTC m=+2455.465279627" watchObservedRunningTime="2026-10-31 15:26:49.849296588 +0000 UTC m=+2455.468848006" Oct 31 15:26:49.964601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:49.964434 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" podStartSLOduration=1.9644119 podStartE2EDuration="1.9644119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:49.959842689 +0000 UTC m=+2455.579394102" watchObservedRunningTime="2026-10-31 15:26:49.9644119 +0000 UTC m=+2455.583963309" Oct 31 15:26:50.241232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:50.241197 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:26:51.193412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:51.193382 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:51.359187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:51.359159 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:26:51.568302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:51.568271 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:52.079985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.079950 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r"] Oct 31 15:26:52.084365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.084348 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.165839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.165800 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:52.167211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.167185 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e72453-openshift-default-dockercfg-x79rl\"" Oct 31 15:26:52.182887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.182864 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r"] Oct 31 15:26:52.263371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263319 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-envoy\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263380 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-data\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263450 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-socket\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263498 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-credential-socket\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263554 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-istio-token\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263597 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55rj\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-kube-api-access-z55rj\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263683 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-certs\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263846 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318ae492-f923-4efd-b588-108ea07bee2f-istio-podinfo\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.263924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.263880 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318ae492-f923-4efd-b588-108ea07bee2f-istiod-ca-cert\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.347564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.347458 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:52.365239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365210 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-certs\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365364 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318ae492-f923-4efd-b588-108ea07bee2f-istio-podinfo\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365403 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318ae492-f923-4efd-b588-108ea07bee2f-istiod-ca-cert\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365529 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-envoy\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365567 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-data\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365639 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-socket\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365663 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-credential-socket\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365719 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-istio-token\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365743 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55rj\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-kube-api-access-z55rj\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.365788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.365763 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-certs\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.366376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.366174 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-data\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.366448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.366378 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-socket\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.366657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.366635 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-credential-socket\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.366910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.366876 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318ae492-f923-4efd-b588-108ea07bee2f-istiod-ca-cert\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.368688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.368668 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318ae492-f923-4efd-b588-108ea07bee2f-istio-podinfo\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.368934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.368918 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-envoy\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.424406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.424375 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-istio-token\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.424406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.424395 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55rj\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-kube-api-access-z55rj\") pod \"e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:52.479217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.479182 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:52.699103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:52.699021 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:53.128933 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:53.128906 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318ae492_f923_4efd_b588_108ea07bee2f.slice/crio-06a9b519f18e0bcf81ab86806f53e9754b21af38f236f91488aa8716e1c1b8df WatchSource:0}: Error finding container 06a9b519f18e0bcf81ab86806f53e9754b21af38f236f91488aa8716e1c1b8df: Status 404 returned error can't find the container with id 06a9b519f18e0bcf81ab86806f53e9754b21af38f236f91488aa8716e1c1b8df Oct 31 15:26:53.131394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:53.131357 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:53.131513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:53.131429 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:53.131513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:53.131460 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:53.132024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:53.132002 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r"] Oct 31 15:26:53.768825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:53.768792 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" event={"ID":"318ae492-f923-4efd-b588-108ea07bee2f","Type":"ContainerStarted","Data":"1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9"} Oct 31 15:26:53.769229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:53.768850 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" event={"ID":"318ae492-f923-4efd-b588-108ea07bee2f","Type":"ContainerStarted","Data":"06a9b519f18e0bcf81ab86806f53e9754b21af38f236f91488aa8716e1c1b8df"} Oct 31 15:26:53.889742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:53.889680 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" podStartSLOduration=1.889661045 podStartE2EDuration="1.889661045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:53.888098447 +0000 UTC m=+2459.507649860" watchObservedRunningTime="2026-10-31 15:26:53.889661045 +0000 UTC m=+2459.509212457" Oct 31 15:26:54.540726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:54.540697 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:54.671777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:54.671742 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:54.699440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:54.699412 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:54.705422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:54.705398 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:54.776306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:54.776274 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:54.777521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:54.777495 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:26:56.453911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:56.453879 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:56.946762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:56.946730 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr"] Oct 31 15:26:56.950792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:56.950762 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:56.952173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:56.952155 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.050372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.050333 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr"] Oct 31 15:26:57.128562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.128511 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.134713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.134680 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.140224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.134985 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.140533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.140507 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.140791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.140770 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.141419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.141396 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.141606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.141588 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.141742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.141723 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.141871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.141856 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscwd\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-kube-api-access-bscwd\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.243750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.243716 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.243951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.243764 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.243951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.243812 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.243951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.243838 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.243951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.243878 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscwd\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-kube-api-access-bscwd\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244033 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244064 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244131 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244172 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244184 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244230 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244265 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244514 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.244924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.244898 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.246336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.246314 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.247121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.247097 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.301742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.301715 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.301888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.301870 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscwd\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-kube-api-access-bscwd\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.440320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.440287 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:57.565880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.565800 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:57.973122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.973083 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:57.998120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:57.998016 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr"] Oct 31 15:26:57.998192 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:57.998147 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3d7b5b_3430_4c12_9a55_58b5723f4c95.slice/crio-42d638a05b2c1f45c7b29a3a0b8d923dd33f8c0b2b82eef290015d75feb0a476 WatchSource:0}: Error finding container 42d638a05b2c1f45c7b29a3a0b8d923dd33f8c0b2b82eef290015d75feb0a476: Status 404 returned error can't find the container with id 42d638a05b2c1f45c7b29a3a0b8d923dd33f8c0b2b82eef290015d75feb0a476 Oct 31 15:26:58.000648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.000614 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:58.000732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.000682 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:58.000732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.000712 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:58.176022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.175989 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:58.365600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.365524 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:58.507495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.505097 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw"] Oct 31 15:26:58.515494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.512639 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.529855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.529776 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:58.661571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.660027 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-21111f-openshift-default-dockercfg-2rl9s\"" Oct 31 15:26:58.755501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.749148 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.773487 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-data\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.773657 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istiod-ca-cert\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.773814 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-podinfo\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.773908 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-envoy\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.773979 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-socket\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.774027 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-credential-socket\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.774101 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-token\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.774287 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crc4m\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-kube-api-access-crc4m\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.774634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.774340 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-certs\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.775594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.775570 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw"] Oct 31 15:26:58.848986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.848112 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" event={"ID":"2e3d7b5b-3430-4c12-9a55-58b5723f4c95","Type":"ContainerStarted","Data":"50b6509da1c676de1791f947bba50902ccc1423ff35d0d1443bf8cd7186daf2f"} Oct 31 15:26:58.848986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.848239 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" event={"ID":"2e3d7b5b-3430-4c12-9a55-58b5723f4c95","Type":"ContainerStarted","Data":"42d638a05b2c1f45c7b29a3a0b8d923dd33f8c0b2b82eef290015d75feb0a476"} Oct 31 15:26:58.875183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875150 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-podinfo\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875200 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-envoy\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875224 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-socket\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875240 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-credential-socket\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875261 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-token\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875396 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crc4m\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-kube-api-access-crc4m\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-certs\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875571 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-data\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.875677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.875648 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istiod-ca-cert\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.876096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.876022 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-certs\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.876096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.876051 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-data\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.876096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.876112 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-socket\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.876422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.876399 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istiod-ca-cert\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.876618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.876594 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-credential-socket\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.878093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.878072 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-podinfo\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.879298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.879277 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-envoy\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.889283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.889253 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:58.937447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.937294 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-token\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.937447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.937355 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crc4m\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-kube-api-access-crc4m\") pod \"e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:58.990799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:58.990751 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podStartSLOduration=2.990738375 podStartE2EDuration="2.990738375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:58.989979225 +0000 UTC m=+2464.609530637" watchObservedRunningTime="2026-10-31 15:26:58.990738375 +0000 UTC m=+2464.610289785" Oct 31 15:26:59.028495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.028246 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:26:59.042818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.042781 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:26:59.525338 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:26:59.525303 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9b2ee2_3aa1_4f79_98d8_cf7690b43851.slice/crio-a5b30f8b8f1e7861b05d78017ea9a89c3417c09359385b97226d77893b15a65f WatchSource:0}: Error finding container a5b30f8b8f1e7861b05d78017ea9a89c3417c09359385b97226d77893b15a65f: Status 404 returned error can't find the container with id a5b30f8b8f1e7861b05d78017ea9a89c3417c09359385b97226d77893b15a65f Oct 31 15:26:59.526672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.526642 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw"] Oct 31 15:26:59.528855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.528816 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:59.528947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.528907 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:59.528986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.528953 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:26:59.566190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.566169 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:26:59.567541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.567520 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 15:26:59.567588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.567569 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 15:26:59.859902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.859792 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" event={"ID":"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851","Type":"ContainerStarted","Data":"d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03"} Oct 31 15:26:59.859902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.859857 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" event={"ID":"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851","Type":"ContainerStarted","Data":"a5b30f8b8f1e7861b05d78017ea9a89c3417c09359385b97226d77893b15a65f"} Oct 31 15:26:59.987024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:26:59.986972 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podStartSLOduration=1.986958778 podStartE2EDuration="1.986958778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:59.986595679 +0000 UTC m=+2465.606147090" watchObservedRunningTime="2026-10-31 15:26:59.986958778 +0000 UTC m=+2465.606510188" Oct 31 15:27:00.043250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.043213 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:27:00.044394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.044361 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 15:27:00.044571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.044411 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 15:27:00.241557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.241527 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:00.566669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.566592 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 15:27:00.566669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.566640 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 15:27:00.743824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.743794 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 15:27:00.744018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:00.743861 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 15:27:01.048642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:01.048612 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:27:01.521542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:01.521507 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:01.566526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:01.566464 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 15:27:01.566677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:01.566545 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 15:27:01.750487 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:01.750436 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:01.879805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:01.879718 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:27:01.881375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:01.881341 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:27:02.125437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.125398 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9"] Oct 31 15:27:02.131027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.130951 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.175889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.174717 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5204b4e7-05da-4091-8727-83cb64939951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.175889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.174956 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqj6s\" (UniqueName: \"kubernetes.io/projected/5204b4e7-05da-4091-8727-83cb64939951-kube-api-access-lqj6s\") pod \"payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.187560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.186820 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-e72453-dockercfg-98q89\"" Oct 31 15:27:02.187560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.186914 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-e72453\"" Oct 31 15:27:02.198497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.195975 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9"] Oct 31 15:27:02.344325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.328051 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqj6s\" (UniqueName: \"kubernetes.io/projected/5204b4e7-05da-4091-8727-83cb64939951-kube-api-access-lqj6s\") pod \"payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.344676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.344646 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5204b4e7-05da-4091-8727-83cb64939951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.346225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.346195 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5204b4e7-05da-4091-8727-83cb64939951-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.362563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.362542 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c"] Oct 31 15:27:02.367155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.367138 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.370233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.369210 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c"] Oct 31 15:27:02.410228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.410158 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqj6s\" (UniqueName: \"kubernetes.io/projected/5204b4e7-05da-4091-8727-83cb64939951-kube-api-access-lqj6s\") pod \"payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.445942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.445911 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf94031-4801-4661-a79a-a80fed8625d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.446075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.445959 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrpvh\" (UniqueName: \"kubernetes.io/projected/ebf94031-4801-4661-a79a-a80fed8625d3-kube-api-access-jrpvh\") pod \"payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.535533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.535493 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:02.547176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.547148 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf94031-4801-4661-a79a-a80fed8625d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.547321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.547209 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrpvh\" (UniqueName: \"kubernetes.io/projected/ebf94031-4801-4661-a79a-a80fed8625d3-kube-api-access-jrpvh\") pod \"payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.547823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.547803 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf94031-4801-4661-a79a-a80fed8625d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.567060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.567031 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 15:27:02.567160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.567082 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 15:27:02.645518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.628279 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrpvh\" (UniqueName: \"kubernetes.io/projected/ebf94031-4801-4661-a79a-a80fed8625d3-kube-api-access-jrpvh\") pod \"payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.685703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.684522 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:02.725507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.724417 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:02.949616 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:27:02.949581 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5204b4e7_05da_4091_8727_83cb64939951.slice/crio-67fb05685654084c03211238fd744edb2eebf0b46da599c502578c79a2b3bd23 WatchSource:0}: Error finding container 67fb05685654084c03211238fd744edb2eebf0b46da599c502578c79a2b3bd23: Status 404 returned error can't find the container with id 67fb05685654084c03211238fd744edb2eebf0b46da599c502578c79a2b3bd23 Oct 31 15:27:02.950514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:02.950487 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9"] Oct 31 15:27:03.122032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.121999 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c"] Oct 31 15:27:03.125963 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:27:03.125937 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf94031_4801_4661_a79a_a80fed8625d3.slice/crio-c21167f6a1fb41db9cca1d52abbd3f1d291e415fefdfdd4a75050f800e5a1f7b WatchSource:0}: Error finding container c21167f6a1fb41db9cca1d52abbd3f1d291e415fefdfdd4a75050f800e5a1f7b: Status 404 returned error can't find the container with id c21167f6a1fb41db9cca1d52abbd3f1d291e415fefdfdd4a75050f800e5a1f7b Oct 31 15:27:03.211894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.211821 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:03.374077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.374043 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:03.518743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.509919 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:03.566292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.566264 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 15:27:03.566522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.566310 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 15:27:03.919966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.919872 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" event={"ID":"ebf94031-4801-4661-a79a-a80fed8625d3","Type":"ContainerStarted","Data":"720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852"} Oct 31 15:27:03.919966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.919925 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" event={"ID":"ebf94031-4801-4661-a79a-a80fed8625d3","Type":"ContainerStarted","Data":"c21167f6a1fb41db9cca1d52abbd3f1d291e415fefdfdd4a75050f800e5a1f7b"} Oct 31 15:27:03.920281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.919970 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:03.921962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.921933 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" event={"ID":"5204b4e7-05da-4091-8727-83cb64939951","Type":"ContainerStarted","Data":"df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079"} Oct 31 15:27:03.922108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.921986 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" event={"ID":"5204b4e7-05da-4091-8727-83cb64939951","Type":"ContainerStarted","Data":"67fb05685654084c03211238fd744edb2eebf0b46da599c502578c79a2b3bd23"} Oct 31 15:27:03.922180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:03.922168 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:04.037252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:04.037197 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" podStartSLOduration=2.037182365 podStartE2EDuration="2.037182365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:04.035737339 +0000 UTC m=+2469.655288751" watchObservedRunningTime="2026-10-31 15:27:04.037182365 +0000 UTC m=+2469.656733775" Oct 31 15:27:04.158145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:04.158070 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" podStartSLOduration=2.158046549 podStartE2EDuration="2.158046549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:04.154636339 +0000 UTC m=+2469.774187750" watchObservedRunningTime="2026-10-31 15:27:04.158046549 +0000 UTC m=+2469.777597966" Oct 31 15:27:04.704519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:04.704453 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:27:04.704987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:04.704965 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:27:04.705843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:04.705826 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:27:05.667910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:05.667829 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:06.429264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:06.429223 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:06.600258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:06.600225 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:08.046126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:08.046097 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:11.518568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:11.518538 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:27:13.387495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:13.387310 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:14.182959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:14.182569 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:27:14.304495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:14.299545 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:27:14.319581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:14.319544 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:27:14.362751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:14.362717 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:27:14.932919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:14.932880 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 15:27:14.933376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:14.932960 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 15:27:17.490204 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:27:17.490163 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:27:18.287911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:18.287881 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:22.421546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:22.421510 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:22.776324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:22.776277 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t"] Oct 31 15:27:22.776623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:22.776593 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" podUID="0b141460-e88a-45b2-b276-140584ae230a" containerName="payload-processing" containerID="cri-o://aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85" gracePeriod=30 Oct 31 15:27:22.827863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:22.827817 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf"] Oct 31 15:27:22.828095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:22.828037 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" podUID="4eac2677-04ee-4ab1-85aa-925e2a239550" containerName="payload-pre-processing" containerID="cri-o://c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05" gracePeriod=30 Oct 31 15:27:22.915381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:22.915347 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:23.036638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.036559 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:23.117680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.117552 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:27:23.122917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.122897 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:27:23.237046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237013 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b141460-e88a-45b2-b276-140584ae230a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b141460-e88a-45b2-b276-140584ae230a-plugins-config-volume\") pod \"0b141460-e88a-45b2-b276-140584ae230a\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " Oct 31 15:27:23.237287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237094 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eac2677-04ee-4ab1-85aa-925e2a239550-kube-api-access-gq48m\" (UniqueName: \"kubernetes.io/projected/4eac2677-04ee-4ab1-85aa-925e2a239550-kube-api-access-gq48m\") pod \"4eac2677-04ee-4ab1-85aa-925e2a239550\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " Oct 31 15:27:23.237287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237205 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eac2677-04ee-4ab1-85aa-925e2a239550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eac2677-04ee-4ab1-85aa-925e2a239550-plugins-config-volume\") pod \"4eac2677-04ee-4ab1-85aa-925e2a239550\" (UID: \"4eac2677-04ee-4ab1-85aa-925e2a239550\") " Oct 31 15:27:23.237405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237298 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b141460-e88a-45b2-b276-140584ae230a-kube-api-access-g9g2g\" (UniqueName: \"kubernetes.io/projected/0b141460-e88a-45b2-b276-140584ae230a-kube-api-access-g9g2g\") pod \"0b141460-e88a-45b2-b276-140584ae230a\" (UID: \"0b141460-e88a-45b2-b276-140584ae230a\") " Oct 31 15:27:23.237852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237484 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b141460-e88a-45b2-b276-140584ae230a-plugins-config-volume" pod "0b141460-e88a-45b2-b276-140584ae230a" (UID: "0b141460-e88a-45b2-b276-140584ae230a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:23.237852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237611 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eac2677-04ee-4ab1-85aa-925e2a239550-plugins-config-volume" pod "4eac2677-04ee-4ab1-85aa-925e2a239550" (UID: "4eac2677-04ee-4ab1-85aa-925e2a239550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:23.238062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237959 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b141460-e88a-45b2-b276-140584ae230a-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:23.238062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.237982 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eac2677-04ee-4ab1-85aa-925e2a239550-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:23.240039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.240008 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b141460-e88a-45b2-b276-140584ae230a-kube-api-access-g9g2g" pod "0b141460-e88a-45b2-b276-140584ae230a" (UID: "0b141460-e88a-45b2-b276-140584ae230a"). InnerVolumeSpecName "kube-api-access-g9g2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:23.240141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.240020 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eac2677-04ee-4ab1-85aa-925e2a239550-kube-api-access-gq48m" pod "4eac2677-04ee-4ab1-85aa-925e2a239550" (UID: "4eac2677-04ee-4ab1-85aa-925e2a239550"). InnerVolumeSpecName "kube-api-access-gq48m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:23.242367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.242344 4865 generic.go:356] "Generic (PLEG): container finished" podID="0b141460-e88a-45b2-b276-140584ae230a" containerID="aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85" exitCode=0 Oct 31 15:27:23.242519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.242402 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" Oct 31 15:27:23.242519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.242426 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" event={"ID":"0b141460-e88a-45b2-b276-140584ae230a","Type":"ContainerDied","Data":"aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85"} Oct 31 15:27:23.242519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.242500 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t" event={"ID":"0b141460-e88a-45b2-b276-140584ae230a","Type":"ContainerDied","Data":"5f9e90de41ecda55ece3ec819b129af9e6ac7cc5f3cdf35af69755d68ed70066"} Oct 31 15:27:23.242639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.242528 4865 scope.go:122] "RemoveContainer" containerID="aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85" Oct 31 15:27:23.244352 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.244334 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" Oct 31 15:27:23.244527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.244343 4865 generic.go:356] "Generic (PLEG): container finished" podID="4eac2677-04ee-4ab1-85aa-925e2a239550" containerID="c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05" exitCode=0 Oct 31 15:27:23.244527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.244511 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" event={"ID":"4eac2677-04ee-4ab1-85aa-925e2a239550","Type":"ContainerDied","Data":"c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05"} Oct 31 15:27:23.244685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.244583 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf" event={"ID":"4eac2677-04ee-4ab1-85aa-925e2a239550","Type":"ContainerDied","Data":"762fb6aa74f403239776e7f7f3953ebc5976739c3b0ba4b7b0aa3df26a505566"} Oct 31 15:27:23.255566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.255548 4865 scope.go:122] "RemoveContainer" containerID="aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85" Oct 31 15:27:23.255881 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:27:23.255858 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85\": container with ID starting with aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85 not found: ID does not exist" containerID="aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85" Oct 31 15:27:23.255980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.255893 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85"} err="failed to get container status \"aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85\": rpc error: code = NotFound desc = could not find container \"aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85\": container with ID starting with aadc80126e0a8a375ac415fdfc339c60881c99617e5f0e87cd35e7797ba83b85 not found: ID does not exist" Oct 31 15:27:23.255980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.255918 4865 scope.go:122] "RemoveContainer" containerID="c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05" Oct 31 15:27:23.269333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.268448 4865 scope.go:122] "RemoveContainer" containerID="c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05" Oct 31 15:27:23.269549 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:27:23.269523 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05\": container with ID starting with c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05 not found: ID does not exist" containerID="c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05" Oct 31 15:27:23.269647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.269565 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05"} err="failed to get container status \"c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05\": rpc error: code = NotFound desc = could not find container \"c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05\": container with ID starting with c3003404bdc2fa83049b9892a1ffae94be61623783ffb7c985fe0c351b3d0f05 not found: ID does not exist" Oct 31 15:27:23.379762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.343945 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq48m\" (UniqueName: \"kubernetes.io/projected/4eac2677-04ee-4ab1-85aa-925e2a239550-kube-api-access-gq48m\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:23.379762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.343985 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9g2g\" (UniqueName: \"kubernetes.io/projected/0b141460-e88a-45b2-b276-140584ae230a-kube-api-access-g9g2g\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:23.490538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.490492 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf"] Oct 31 15:27:23.494987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.494965 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4ede8-64c968d4d7-hprgf"] Oct 31 15:27:23.606535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.606500 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t"] Oct 31 15:27:23.616451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:23.616409 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4ede8-5cd89f978-lrm6t"] Oct 31 15:27:24.193442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:24.193354 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:24.497433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:24.497401 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b141460-e88a-45b2-b276-140584ae230a" path="/var/lib/kubelet/pods/0b141460-e88a-45b2-b276-140584ae230a/volumes" Oct 31 15:27:24.497918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:24.497857 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eac2677-04ee-4ab1-85aa-925e2a239550" path="/var/lib/kubelet/pods/4eac2677-04ee-4ab1-85aa-925e2a239550/volumes" Oct 31 15:27:28.015551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.015517 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:28.056575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.056538 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7"] Oct 31 15:27:28.057226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.057210 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0b141460-e88a-45b2-b276-140584ae230a" containerName="payload-processing" Oct 31 15:27:28.057282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.057229 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b141460-e88a-45b2-b276-140584ae230a" containerName="payload-processing" Oct 31 15:27:28.057282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.057252 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4eac2677-04ee-4ab1-85aa-925e2a239550" containerName="payload-pre-processing" Oct 31 15:27:28.057282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.057258 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eac2677-04ee-4ab1-85aa-925e2a239550" containerName="payload-pre-processing" Oct 31 15:27:28.057641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.057618 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b141460-e88a-45b2-b276-140584ae230a" containerName="payload-processing" Oct 31 15:27:28.057743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.057730 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eac2677-04ee-4ab1-85aa-925e2a239550" containerName="payload-pre-processing" Oct 31 15:27:28.064107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.064072 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.112286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.112257 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7"] Oct 31 15:27:28.136208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.136177 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:28.158283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.158255 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:27:28.158283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.158255 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:27:28.158486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.158262 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:27:28.168319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.168298 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:27:28.208434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.208412 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.208554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.208503 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng42q\" (UniqueName: \"kubernetes.io/projected/a3b7a2e9-7541-4540-81da-90e62cdf2409-kube-api-access-ng42q\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.208684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.208671 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.310502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.310403 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.310502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.310463 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng42q\" (UniqueName: \"kubernetes.io/projected/a3b7a2e9-7541-4540-81da-90e62cdf2409-kube-api-access-ng42q\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.310738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.310632 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.311308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.311285 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.313176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.313153 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.367995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.367969 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng42q\" (UniqueName: \"kubernetes.io/projected/a3b7a2e9-7541-4540-81da-90e62cdf2409-kube-api-access-ng42q\") pod \"maas-api-revoke-sub-01328dd6-k2cv7\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.382462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.382438 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:28.761886 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:27:28.761852 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b7a2e9_7541_4540_81da_90e62cdf2409.slice/crio-132a0260b446ef2e4d09c0421eab180ca6e8971519eb5f690abb9f3baa1edb0e WatchSource:0}: Error finding container 132a0260b446ef2e4d09c0421eab180ca6e8971519eb5f690abb9f3baa1edb0e: Status 404 returned error can't find the container with id 132a0260b446ef2e4d09c0421eab180ca6e8971519eb5f690abb9f3baa1edb0e Oct 31 15:27:28.763628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.763523 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7"] Oct 31 15:27:28.896729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:28.896698 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:29.053054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:29.053022 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:29.175092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:29.175054 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:29.299129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:29.299041 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" event={"ID":"a3b7a2e9-7541-4540-81da-90e62cdf2409","Type":"ContainerStarted","Data":"72dbf68ee91282e96d772f160937cec4a7854eb2e52f4ef6000b8d5dbca101a1"} Oct 31 15:27:29.299129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:29.299113 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" event={"ID":"a3b7a2e9-7541-4540-81da-90e62cdf2409","Type":"ContainerStarted","Data":"132a0260b446ef2e4d09c0421eab180ca6e8971519eb5f690abb9f3baa1edb0e"} Oct 31 15:27:29.418884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:29.418820 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" podStartSLOduration=1.418802974 podStartE2EDuration="1.418802974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:29.41397773 +0000 UTC m=+2495.033529140" watchObservedRunningTime="2026-10-31 15:27:29.418802974 +0000 UTC m=+2495.038354386" Oct 31 15:27:30.309026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:30.308994 4865 generic.go:356] "Generic (PLEG): container finished" podID="a3b7a2e9-7541-4540-81da-90e62cdf2409" containerID="72dbf68ee91282e96d772f160937cec4a7854eb2e52f4ef6000b8d5dbca101a1" exitCode=0 Oct 31 15:27:30.309432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:30.309073 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" event={"ID":"a3b7a2e9-7541-4540-81da-90e62cdf2409","Type":"ContainerDied","Data":"72dbf68ee91282e96d772f160937cec4a7854eb2e52f4ef6000b8d5dbca101a1"} Oct 31 15:27:30.698870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:30.698793 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:30.834507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:30.834446 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:31.486059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.486035 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:31.552983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.552951 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-xks-ca\") pod \"a3b7a2e9-7541-4540-81da-90e62cdf2409\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " Oct 31 15:27:31.553150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.553050 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b7a2e9-7541-4540-81da-90e62cdf2409-kube-api-access-ng42q\" (UniqueName: \"kubernetes.io/projected/a3b7a2e9-7541-4540-81da-90e62cdf2409-kube-api-access-ng42q\") pod \"a3b7a2e9-7541-4540-81da-90e62cdf2409\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " Oct 31 15:27:31.553220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.553200 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-openshift-ca\") pod \"a3b7a2e9-7541-4540-81da-90e62cdf2409\" (UID: \"a3b7a2e9-7541-4540-81da-90e62cdf2409\") " Oct 31 15:27:31.553610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.553542 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-openshift-ca" pod "a3b7a2e9-7541-4540-81da-90e62cdf2409" (UID: "a3b7a2e9-7541-4540-81da-90e62cdf2409"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:31.554066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.554042 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:31.555206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.555183 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-xks-ca" pod "a3b7a2e9-7541-4540-81da-90e62cdf2409" (UID: "a3b7a2e9-7541-4540-81da-90e62cdf2409"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:27:31.555314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.555299 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b7a2e9-7541-4540-81da-90e62cdf2409-kube-api-access-ng42q" pod "a3b7a2e9-7541-4540-81da-90e62cdf2409" (UID: "a3b7a2e9-7541-4540-81da-90e62cdf2409"). InnerVolumeSpecName "kube-api-access-ng42q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:31.655685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.655615 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b7a2e9-7541-4540-81da-90e62cdf2409-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:31.655685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:31.655647 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng42q\" (UniqueName: \"kubernetes.io/projected/a3b7a2e9-7541-4540-81da-90e62cdf2409-kube-api-access-ng42q\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:32.185635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:32.185593 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:32.326852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:32.326818 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" Oct 31 15:27:32.327043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:32.326818 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7" event={"ID":"a3b7a2e9-7541-4540-81da-90e62cdf2409","Type":"ContainerDied","Data":"132a0260b446ef2e4d09c0421eab180ca6e8971519eb5f690abb9f3baa1edb0e"} Oct 31 15:27:32.327043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:32.326969 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="132a0260b446ef2e4d09c0421eab180ca6e8971519eb5f690abb9f3baa1edb0e" Oct 31 15:27:32.615060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:32.615027 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv"] Oct 31 15:27:32.615615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:32.615259 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" containerID="cri-o://03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24" gracePeriod=30 Oct 31 15:27:32.645017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:32.644987 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:33.456068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:33.456032 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8"] Oct 31 15:27:33.567046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:33.567010 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b"] Oct 31 15:27:33.681207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:33.681156 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:33.838396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:33.838357 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:33.966219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:33.966174 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:34.096889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.096798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:34.311781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.311750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:34.762616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.762588 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv"] Oct 31 15:27:34.763225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.763211 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a3b7a2e9-7541-4540-81da-90e62cdf2409" containerName="revoke-keys" Oct 31 15:27:34.763275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.763226 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b7a2e9-7541-4540-81da-90e62cdf2409" containerName="revoke-keys" Oct 31 15:27:34.763342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.763332 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b7a2e9-7541-4540-81da-90e62cdf2409" containerName="revoke-keys" Oct 31 15:27:34.767193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.767179 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.825295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.825266 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv"] Oct 31 15:27:34.848944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.848916 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:34.870815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.870785 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5fccd7-openshift-default-dockercfg-hf2cr\"" Oct 31 15:27:34.963496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.929401 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-podinfo\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.963933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.963902 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-token\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.964246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.964224 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-envoy\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.964387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.964370 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-socket\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.964536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.964517 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxh5\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-kube-api-access-nfxh5\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.964718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.964697 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-credential-socket\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.964848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.964833 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istiod-ca-cert\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.964953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.964938 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-certs\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:34.987495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:34.978708 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-data\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.079916 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-token\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080075 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-envoy\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080122 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-socket\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080156 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxh5\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-kube-api-access-nfxh5\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080212 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-credential-socket\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080254 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istiod-ca-cert\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080284 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-certs\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080439 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-data\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080491 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-podinfo\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080709 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-credential-socket\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080749 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-socket\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.080936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.080837 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-certs\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.081210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.081183 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-data\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.081995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.081967 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istiod-ca-cert\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.083394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.083375 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-envoy\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.087036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.087009 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-podinfo\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.138424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.138395 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxh5\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-kube-api-access-nfxh5\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.144957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.144926 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-token\") pod \"e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.215489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.215437 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:35.382138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.382045 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:35.829005 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:27:35.828967 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb908f7b_125c_4283_a4cd_09e603a2c4d9.slice/crio-a68a28dfe5c709b39fec3a88492b7028bc092c1e8ab91207c6d799c44f277371 WatchSource:0}: Error finding container a68a28dfe5c709b39fec3a88492b7028bc092c1e8ab91207c6d799c44f277371: Status 404 returned error can't find the container with id a68a28dfe5c709b39fec3a88492b7028bc092c1e8ab91207c6d799c44f277371 Oct 31 15:27:35.830458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.830434 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv"] Oct 31 15:27:35.837384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.832875 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:27:35.837384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.833115 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:27:35.837384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:35.833334 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:27:36.361041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:36.361003 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" event={"ID":"eb908f7b-125c-4283-a4cd-09e603a2c4d9","Type":"ContainerStarted","Data":"b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f"} Oct 31 15:27:36.361231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:36.361089 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" event={"ID":"eb908f7b-125c-4283-a4cd-09e603a2c4d9","Type":"ContainerStarted","Data":"a68a28dfe5c709b39fec3a88492b7028bc092c1e8ab91207c6d799c44f277371"} Oct 31 15:27:36.382310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:36.382286 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:36.387567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:36.387549 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:36.550727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:36.550676 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" podStartSLOduration=2.550662675 podStartE2EDuration="2.550662675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:36.529196314 +0000 UTC m=+2502.148747725" watchObservedRunningTime="2026-10-31 15:27:36.550662675 +0000 UTC m=+2502.170214085" Oct 31 15:27:37.368812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:37.368786 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:37.369708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:37.369687 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:27:37.887205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:37.887135 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:27:38.017217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017186 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-certs\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017298 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-podinfo\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017356 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-socket\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017419 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f131f3a-4de2-42b9-b875-f0d007a674fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f131f3a-4de2-42b9-b875-f0d007a674fe-istiod-ca-cert\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017455 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-kube-api-access-pnqfh\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-kube-api-access-pnqfh\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017516 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-certs" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:38.017568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017531 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-envoy\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017594 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-credential-socket\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017695 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-token\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-token\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017694 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-socket" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:38.017972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017805 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-data\") pod \"0f131f3a-4de2-42b9-b875-f0d007a674fe\" (UID: \"0f131f3a-4de2-42b9-b875-f0d007a674fe\") " Oct 31 15:27:38.017972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.017882 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-credential-socket" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:38.018155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.018117 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f131f3a-4de2-42b9-b875-f0d007a674fe-istiod-ca-cert" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:38.018712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.018606 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.018712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.018634 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.018712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.018649 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f131f3a-4de2-42b9-b875-f0d007a674fe-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.018712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.018653 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-data" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:38.018712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.018665 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.021912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.021883 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-kube-api-access-pnqfh" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "kube-api-access-pnqfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:38.021912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.021893 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-podinfo" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:27:38.022060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.022026 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-envoy" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:38.022131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.022117 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-token" pod "0f131f3a-4de2-42b9-b875-f0d007a674fe" (UID: "0f131f3a-4de2-42b9-b875-f0d007a674fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:38.120119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.120087 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqfh\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-kube-api-access-pnqfh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.120119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.120117 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.120119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.120126 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.120370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.120134 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.120370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.120143 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f131f3a-4de2-42b9-b875-f0d007a674fe-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:38.379048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.379015 4865 generic.go:356] "Generic (PLEG): container finished" podID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerID="03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24" exitCode=0 Oct 31 15:27:38.379600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.379076 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" Oct 31 15:27:38.379600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.379095 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" event={"ID":"0f131f3a-4de2-42b9-b875-f0d007a674fe","Type":"ContainerDied","Data":"03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24"} Oct 31 15:27:38.379600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.379146 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv" event={"ID":"0f131f3a-4de2-42b9-b875-f0d007a674fe","Type":"ContainerDied","Data":"b3b95557e81b8c93d888f8c6ca79e1920ba77fcf84ed5f9c4f84e60264dabc3d"} Oct 31 15:27:38.379600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.379169 4865 scope.go:122] "RemoveContainer" containerID="03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24" Oct 31 15:27:38.390688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.390669 4865 scope.go:122] "RemoveContainer" containerID="9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd" Oct 31 15:27:38.400684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.400656 4865 scope.go:122] "RemoveContainer" containerID="03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24" Oct 31 15:27:38.400941 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:27:38.400924 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24\": container with ID starting with 03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24 not found: ID does not exist" containerID="03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24" Oct 31 15:27:38.400994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.400951 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24"} err="failed to get container status \"03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24\": rpc error: code = NotFound desc = could not find container \"03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24\": container with ID starting with 03a126368ed18b073b93a4fdda8de153f8977d161be41eb7f22ddabca119fd24 not found: ID does not exist" Oct 31 15:27:38.400994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.400968 4865 scope.go:122] "RemoveContainer" containerID="9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd" Oct 31 15:27:38.401203 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:27:38.401182 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd\": container with ID starting with 9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd not found: ID does not exist" containerID="9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd" Oct 31 15:27:38.401247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.401211 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd"} err="failed to get container status \"9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd\": rpc error: code = NotFound desc = could not find container \"9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd\": container with ID starting with 9d6a4d5382c7c8a9e5a3782bd4cea1565d7a8fe8d5808a0ade01b7d97d5a2edd not found: ID does not exist" Oct 31 15:27:38.456598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.456561 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" podUID="40d70601-d132-4771-a7a5-f9e9080ac2bb" containerName="payload-processing" containerID="cri-o://51ac34c4b8c3f45df60a78cacd1eff49b33fa6960ae5feb4d11795c53ee760f1" gracePeriod=25 Oct 31 15:27:38.548298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.548266 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv"] Oct 31 15:27:38.568026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.567996 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" podUID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" containerName="payload-pre-processing" containerID="cri-o://d2b50fc65cf831f1cf84a99982c7c196b5b31e688949e9efc9b75d1276939689" gracePeriod=25 Oct 31 15:27:38.599176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:38.599149 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c4ede8-openshift-default-779fb6b5dc-pjlfv"] Oct 31 15:27:39.273238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:39.273209 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:39.441939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:39.441905 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:39.593259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:39.593181 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:39.816807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:39.816760 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:39.968366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:39.968337 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:40.114099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:40.114061 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:40.497259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:40.497219 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" path="/var/lib/kubelet/pods/0f131f3a-4de2-42b9-b875-f0d007a674fe/volumes" Oct 31 15:27:40.687839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:40.687810 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:42.855578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:42.855535 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" podUID="40d70601-d132-4771-a7a5-f9e9080ac2bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:27:43.019348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.019311 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:43.188108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.186413 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:43.430198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.430158 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g"] Oct 31 15:27:43.431186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.431160 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" Oct 31 15:27:43.431186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.431184 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" Oct 31 15:27:43.431403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.431207 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" Oct 31 15:27:43.431403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.431212 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" Oct 31 15:27:43.431403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.431329 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" Oct 31 15:27:43.431403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.431339 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f131f3a-4de2-42b9-b875-f0d007a674fe" containerName="istio-proxy" Oct 31 15:27:43.436819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.436800 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.479606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.479577 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g"] Oct 31 15:27:43.550655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.550616 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfvk\" (UniqueName: \"kubernetes.io/projected/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-kube-api-access-wwfvk\") pod \"payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.569515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.550970 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.584168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.584133 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5fccd7\"" Oct 31 15:27:43.584398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.584251 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5fccd7-dockercfg-lmtpt\"" Oct 31 15:27:43.672270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.671890 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfvk\" (UniqueName: \"kubernetes.io/projected/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-kube-api-access-wwfvk\") pod \"payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.672270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.671965 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.673052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.673018 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.692641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.683604 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw"] Oct 31 15:27:43.717546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.699661 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw"] Oct 31 15:27:43.717546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.699779 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:43.733526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.733492 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfvk\" (UniqueName: \"kubernetes.io/projected/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-kube-api-access-wwfvk\") pod \"payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.750110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.750079 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:43.862599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.862551 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" podUID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:27:43.875341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.875310 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14afe227-bfaf-41f9-8d70-e537d851dd57-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:43.875511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.875453 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nsgr\" (UniqueName: \"kubernetes.io/projected/14afe227-bfaf-41f9-8d70-e537d851dd57-kube-api-access-7nsgr\") pod \"payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:43.977042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.976947 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14afe227-bfaf-41f9-8d70-e537d851dd57-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:43.977211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.977145 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nsgr\" (UniqueName: \"kubernetes.io/projected/14afe227-bfaf-41f9-8d70-e537d851dd57-kube-api-access-7nsgr\") pod \"payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:43.977752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:43.977731 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14afe227-bfaf-41f9-8d70-e537d851dd57-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:44.035144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.035106 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nsgr\" (UniqueName: \"kubernetes.io/projected/14afe227-bfaf-41f9-8d70-e537d851dd57-kube-api-access-7nsgr\") pod \"payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:44.055001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.054971 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:44.166289 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:27:44.166254 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e2f292_95c5_4e25_9e3c_c64ae7cd0891.slice/crio-04db2a13dacb6821422d141f7ff95c4367198aaf7a599d6bf3f1dd1bdfb70358 WatchSource:0}: Error finding container 04db2a13dacb6821422d141f7ff95c4367198aaf7a599d6bf3f1dd1bdfb70358: Status 404 returned error can't find the container with id 04db2a13dacb6821422d141f7ff95c4367198aaf7a599d6bf3f1dd1bdfb70358 Oct 31 15:27:44.167260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.167216 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g"] Oct 31 15:27:44.452422 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:27:44.452394 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14afe227_bfaf_41f9_8d70_e537d851dd57.slice/crio-e34aebb59bb19e5295eb97254ee8b3fb0202ca9d128484b5ec2d2ccb33522824 WatchSource:0}: Error finding container e34aebb59bb19e5295eb97254ee8b3fb0202ca9d128484b5ec2d2ccb33522824: Status 404 returned error can't find the container with id e34aebb59bb19e5295eb97254ee8b3fb0202ca9d128484b5ec2d2ccb33522824 Oct 31 15:27:44.453671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.453646 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw"] Oct 31 15:27:44.547435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.547405 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:44.592664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.592629 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" event={"ID":"47e2f292-95c5-4e25-9e3c-c64ae7cd0891","Type":"ContainerStarted","Data":"585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1"} Oct 31 15:27:44.592797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.592693 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" event={"ID":"47e2f292-95c5-4e25-9e3c-c64ae7cd0891","Type":"ContainerStarted","Data":"04db2a13dacb6821422d141f7ff95c4367198aaf7a599d6bf3f1dd1bdfb70358"} Oct 31 15:27:44.592797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.592722 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:44.621495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.621156 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" event={"ID":"14afe227-bfaf-41f9-8d70-e537d851dd57","Type":"ContainerStarted","Data":"8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9"} Oct 31 15:27:44.621495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.621223 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" event={"ID":"14afe227-bfaf-41f9-8d70-e537d851dd57","Type":"ContainerStarted","Data":"e34aebb59bb19e5295eb97254ee8b3fb0202ca9d128484b5ec2d2ccb33522824"} Oct 31 15:27:44.626496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.622104 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:44.722685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.722619 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" podStartSLOduration=1.722603271 podStartE2EDuration="1.722603271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:44.718890859 +0000 UTC m=+2510.338442271" watchObservedRunningTime="2026-10-31 15:27:44.722603271 +0000 UTC m=+2510.342154767" Oct 31 15:27:44.838708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:44.838454 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" podStartSLOduration=1.838432605 podStartE2EDuration="1.838432605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:44.834843331 +0000 UTC m=+2510.454394744" watchObservedRunningTime="2026-10-31 15:27:44.838432605 +0000 UTC m=+2510.457984017" Oct 31 15:27:47.495610 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:27:47.495571 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:27:49.458421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:49.458364 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:49.627292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:49.627253 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:49.809153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:49.809118 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:50.671999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:50.671960 4865 generic.go:356] "Generic (PLEG): container finished" podID="924114e3-1c2c-432e-8492-b7f6811359ea" containerID="b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605" exitCode=0 Oct 31 15:27:50.672549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:50.672007 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" event={"ID":"924114e3-1c2c-432e-8492-b7f6811359ea","Type":"ContainerDied","Data":"b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605"} Oct 31 15:27:50.672639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:50.672603 4865 scope.go:122] "RemoveContainer" containerID="b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605" Oct 31 15:27:50.674281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:50.674233 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:27:50.674503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:50.674340 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:27:50.674503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:50.674402 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:27:51.693101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:51.693064 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" event={"ID":"924114e3-1c2c-432e-8492-b7f6811359ea","Type":"ContainerStarted","Data":"11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685"} Oct 31 15:27:51.748207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:51.748170 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:27:51.749087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:51.749059 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 15:27:51.749214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:51.749140 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 15:27:52.747550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:52.747516 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 15:27:52.747992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:52.747592 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 15:27:52.855899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:52.855862 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" podUID="40d70601-d132-4771-a7a5-f9e9080ac2bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:27:53.747562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:53.747524 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 15:27:53.748001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:53.747595 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 15:27:53.862210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:53.862171 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" podUID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:27:54.310623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:54.310586 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:54.747648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:54.747615 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 15:27:54.748084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:54.747674 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 15:27:55.239898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:55.239869 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:27:55.632689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:55.632592 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:27:55.747377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:55.747341 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 31 15:27:55.747611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:55.747401 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 31 15:27:56.641578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:56.641546 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:27:56.806222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:56.806184 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:27:56.809827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:56.809800 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 15:27:58.771002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.770973 4865 generic.go:356] "Generic (PLEG): container finished" podID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" containerID="d2b50fc65cf831f1cf84a99982c7c196b5b31e688949e9efc9b75d1276939689" exitCode=0 Oct 31 15:27:58.771372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.771047 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" event={"ID":"cc6f8385-144a-44f8-98fb-e0bec936f0ff","Type":"ContainerDied","Data":"d2b50fc65cf831f1cf84a99982c7c196b5b31e688949e9efc9b75d1276939689"} Oct 31 15:27:58.772558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.772534 4865 generic.go:356] "Generic (PLEG): container finished" podID="40d70601-d132-4771-a7a5-f9e9080ac2bb" containerID="51ac34c4b8c3f45df60a78cacd1eff49b33fa6960ae5feb4d11795c53ee760f1" exitCode=0 Oct 31 15:27:58.772666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.772604 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" event={"ID":"40d70601-d132-4771-a7a5-f9e9080ac2bb","Type":"ContainerDied","Data":"51ac34c4b8c3f45df60a78cacd1eff49b33fa6960ae5feb4d11795c53ee760f1"} Oct 31 15:27:58.772666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.772646 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" event={"ID":"40d70601-d132-4771-a7a5-f9e9080ac2bb","Type":"ContainerDied","Data":"ed1dee29eeef826670225f46142b69a897bbfcaf42e9eb97a2db0487debadc07"} Oct 31 15:27:58.772666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.772662 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed1dee29eeef826670225f46142b69a897bbfcaf42e9eb97a2db0487debadc07" Oct 31 15:27:58.773238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.773227 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:27:58.865646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.865623 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:27:58.957829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.957795 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d70601-d132-4771-a7a5-f9e9080ac2bb-kube-api-access-rl6zb\" (UniqueName: \"kubernetes.io/projected/40d70601-d132-4771-a7a5-f9e9080ac2bb-kube-api-access-rl6zb\") pod \"40d70601-d132-4771-a7a5-f9e9080ac2bb\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " Oct 31 15:27:58.958025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.957964 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d70601-d132-4771-a7a5-f9e9080ac2bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70601-d132-4771-a7a5-f9e9080ac2bb-plugins-config-volume\") pod \"40d70601-d132-4771-a7a5-f9e9080ac2bb\" (UID: \"40d70601-d132-4771-a7a5-f9e9080ac2bb\") " Oct 31 15:27:58.958391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.958368 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d70601-d132-4771-a7a5-f9e9080ac2bb-plugins-config-volume" pod "40d70601-d132-4771-a7a5-f9e9080ac2bb" (UID: "40d70601-d132-4771-a7a5-f9e9080ac2bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:58.958596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.958581 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d70601-d132-4771-a7a5-f9e9080ac2bb-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:58.960042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:58.960021 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d70601-d132-4771-a7a5-f9e9080ac2bb-kube-api-access-rl6zb" pod "40d70601-d132-4771-a7a5-f9e9080ac2bb" (UID: "40d70601-d132-4771-a7a5-f9e9080ac2bb"). InnerVolumeSpecName "kube-api-access-rl6zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:59.059668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.059579 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6f8385-144a-44f8-98fb-e0bec936f0ff-kube-api-access-lj5dr\" (UniqueName: \"kubernetes.io/projected/cc6f8385-144a-44f8-98fb-e0bec936f0ff-kube-api-access-lj5dr\") pod \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " Oct 31 15:27:59.059668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.059646 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6f8385-144a-44f8-98fb-e0bec936f0ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f8385-144a-44f8-98fb-e0bec936f0ff-plugins-config-volume\") pod \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\" (UID: \"cc6f8385-144a-44f8-98fb-e0bec936f0ff\") " Oct 31 15:27:59.060058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.060031 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6f8385-144a-44f8-98fb-e0bec936f0ff-plugins-config-volume" pod "cc6f8385-144a-44f8-98fb-e0bec936f0ff" (UID: "cc6f8385-144a-44f8-98fb-e0bec936f0ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:59.060252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.060239 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6zb\" (UniqueName: \"kubernetes.io/projected/40d70601-d132-4771-a7a5-f9e9080ac2bb-kube-api-access-rl6zb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:59.060302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.060255 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6f8385-144a-44f8-98fb-e0bec936f0ff-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:59.061940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.061922 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6f8385-144a-44f8-98fb-e0bec936f0ff-kube-api-access-lj5dr" pod "cc6f8385-144a-44f8-98fb-e0bec936f0ff" (UID: "cc6f8385-144a-44f8-98fb-e0bec936f0ff"). InnerVolumeSpecName "kube-api-access-lj5dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:59.161542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.161500 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5dr\" (UniqueName: \"kubernetes.io/projected/cc6f8385-144a-44f8-98fb-e0bec936f0ff-kube-api-access-lj5dr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:59.214872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.214834 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:59.339111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.339012 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:27:59.783075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.783049 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" Oct 31 15:27:59.783533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.783078 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b" event={"ID":"cc6f8385-144a-44f8-98fb-e0bec936f0ff","Type":"ContainerDied","Data":"b95fa5145b99046fa6dd67c7fe03564564cc0e058c7292f7fbd2b8bdec594351"} Oct 31 15:27:59.783533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.783137 4865 scope.go:122] "RemoveContainer" containerID="d2b50fc65cf831f1cf84a99982c7c196b5b31e688949e9efc9b75d1276939689" Oct 31 15:27:59.783533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.783315 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8" Oct 31 15:27:59.951976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.951950 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8"] Oct 31 15:27:59.955936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:27:59.955917 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d9cddd-8557ff6f67-6fgc8"] Oct 31 15:28:00.066366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:00.066286 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b"] Oct 31 15:28:00.070595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:00.070569 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d9cddd-8f56f5f6c-rjp9b"] Oct 31 15:28:00.498214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:00.498175 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d70601-d132-4771-a7a5-f9e9080ac2bb" path="/var/lib/kubelet/pods/40d70601-d132-4771-a7a5-f9e9080ac2bb/volumes" Oct 31 15:28:00.498639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:00.498624 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" path="/var/lib/kubelet/pods/cc6f8385-144a-44f8-98fb-e0bec936f0ff/volumes" Oct 31 15:28:00.582273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:00.582207 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:03.948962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:03.948924 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:07.446352 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:07.444612 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:28:08.711286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:08.710892 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:13.048324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:13.048274 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4"] Oct 31 15:28:13.133699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:13.133663 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc465a6-vd6v4"] Oct 31 15:28:13.393562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:13.393436 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:14.520071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:14.517596 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e20d64-be4a-4274-84d8-d8ee73e6a092" path="/var/lib/kubelet/pods/a0e20d64-be4a-4274-84d8-d8ee73e6a092/volumes" Oct 31 15:28:16.799774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.799733 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln"] Oct 31 15:28:16.800404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.800380 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" containerName="payload-pre-processing" Oct 31 15:28:16.800404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.800392 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" containerName="payload-pre-processing" Oct 31 15:28:16.800574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.800426 4865 cpu_manager.go:414] "containerMap: removing container" podUID="40d70601-d132-4771-a7a5-f9e9080ac2bb" containerName="payload-processing" Oct 31 15:28:16.800574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.800431 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d70601-d132-4771-a7a5-f9e9080ac2bb" containerName="payload-processing" Oct 31 15:28:16.800574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.800540 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d70601-d132-4771-a7a5-f9e9080ac2bb" containerName="payload-processing" Oct 31 15:28:16.800574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.800554 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6f8385-144a-44f8-98fb-e0bec936f0ff" containerName="payload-pre-processing" Oct 31 15:28:16.804374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.804355 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.860083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.860048 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln"] Oct 31 15:28:16.890953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.890907 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.891157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.891090 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.891231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.891152 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42dd\" (UniqueName: \"kubernetes.io/projected/6c6484b6-f7d6-4308-9cec-5631de746ee1-kube-api-access-b42dd\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.905039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.905014 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:28:16.905494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.905435 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:28:16.905686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.905653 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:28:16.908752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.908732 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:28:16.992745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.992714 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.992927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.992791 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42dd\" (UniqueName: \"kubernetes.io/projected/6c6484b6-f7d6-4308-9cec-5631de746ee1-kube-api-access-b42dd\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.992997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.992963 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.993626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.993569 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:16.995365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:16.995341 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:17.050829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:17.050766 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42dd\" (UniqueName: \"kubernetes.io/projected/6c6484b6-f7d6-4308-9cec-5631de746ee1-kube-api-access-b42dd\") pod \"maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:17.119016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:17.118978 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:17.491879 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:28:17.491838 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:28:17.504246 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:28:17.504217 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6484b6_f7d6_4308_9cec_5631de746ee1.slice/crio-31f1931f2a8369f46466d6a265db6f02d25105cb4207a283d66d0310885fe407 WatchSource:0}: Error finding container 31f1931f2a8369f46466d6a265db6f02d25105cb4207a283d66d0310885fe407: Status 404 returned error can't find the container with id 31f1931f2a8369f46466d6a265db6f02d25105cb4207a283d66d0310885fe407 Oct 31 15:28:17.510090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:17.506733 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln"] Oct 31 15:28:17.920631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:17.920593 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:28:17.921164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:17.920649 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:28:17.927531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:17.927500 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" event={"ID":"6c6484b6-f7d6-4308-9cec-5631de746ee1","Type":"ContainerStarted","Data":"95d58c967483aa6d6151507b0098223d782a093a3dbb7291f362e035b877c720"} Oct 31 15:28:17.927711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:17.927565 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" event={"ID":"6c6484b6-f7d6-4308-9cec-5631de746ee1","Type":"ContainerStarted","Data":"31f1931f2a8369f46466d6a265db6f02d25105cb4207a283d66d0310885fe407"} Oct 31 15:28:18.046954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.046895 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" podStartSLOduration=2.046880756 podStartE2EDuration="2.046880756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:18.041960794 +0000 UTC m=+2543.661512206" watchObservedRunningTime="2026-10-31 15:28:18.046880756 +0000 UTC m=+2543.666432215" Oct 31 15:28:18.293139 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.293106 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:18.937456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.937381 4865 generic.go:356] "Generic (PLEG): container finished" podID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerID="d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03" exitCode=0 Oct 31 15:28:18.937919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.937446 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" event={"ID":"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851","Type":"ContainerDied","Data":"d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03"} Oct 31 15:28:18.937919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.937878 4865 scope.go:122] "RemoveContainer" containerID="d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03" Oct 31 15:28:18.939402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.939370 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:18.939549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.939421 4865 generic.go:356] "Generic (PLEG): container finished" podID="318ae492-f923-4efd-b588-108ea07bee2f" containerID="1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9" exitCode=0 Oct 31 15:28:18.939549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.939440 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:18.939549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.939505 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:18.939703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.939502 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" event={"ID":"318ae492-f923-4efd-b588-108ea07bee2f","Type":"ContainerDied","Data":"1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9"} Oct 31 15:28:18.940010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.939935 4865 scope.go:122] "RemoveContainer" containerID="1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9" Oct 31 15:28:18.941379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.941319 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:18.941522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.941402 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:18.941522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.941452 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:18.941651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.941521 4865 generic.go:356] "Generic (PLEG): container finished" podID="6c6484b6-f7d6-4308-9cec-5631de746ee1" containerID="95d58c967483aa6d6151507b0098223d782a093a3dbb7291f362e035b877c720" exitCode=0 Oct 31 15:28:18.941651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:18.941564 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" event={"ID":"6c6484b6-f7d6-4308-9cec-5631de746ee1","Type":"ContainerDied","Data":"95d58c967483aa6d6151507b0098223d782a093a3dbb7291f362e035b877c720"} Oct 31 15:28:19.958118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:19.958082 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" event={"ID":"318ae492-f923-4efd-b588-108ea07bee2f","Type":"ContainerStarted","Data":"048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5"} Oct 31 15:28:19.961173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:19.961143 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" event={"ID":"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851","Type":"ContainerStarted","Data":"4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a"} Oct 31 15:28:20.046158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.046111 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 15:28:20.046373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.046173 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 15:28:20.046554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.046525 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:28:20.252779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.252713 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:20.360246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.360164 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-openshift-ca\") pod \"6c6484b6-f7d6-4308-9cec-5631de746ee1\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " Oct 31 15:28:20.360445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.360382 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6484b6-f7d6-4308-9cec-5631de746ee1-kube-api-access-b42dd\" (UniqueName: \"kubernetes.io/projected/6c6484b6-f7d6-4308-9cec-5631de746ee1-kube-api-access-b42dd\") pod \"6c6484b6-f7d6-4308-9cec-5631de746ee1\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " Oct 31 15:28:20.360544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.360515 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-xks-ca\") pod \"6c6484b6-f7d6-4308-9cec-5631de746ee1\" (UID: \"6c6484b6-f7d6-4308-9cec-5631de746ee1\") " Oct 31 15:28:20.360665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.360644 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-openshift-ca" pod "6c6484b6-f7d6-4308-9cec-5631de746ee1" (UID: "6c6484b6-f7d6-4308-9cec-5631de746ee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:28:20.361115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.360971 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:28:20.363009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.362980 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-xks-ca" pod "6c6484b6-f7d6-4308-9cec-5631de746ee1" (UID: "6c6484b6-f7d6-4308-9cec-5631de746ee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:28:20.363009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.362989 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6484b6-f7d6-4308-9cec-5631de746ee1-kube-api-access-b42dd" pod "6c6484b6-f7d6-4308-9cec-5631de746ee1" (UID: "6c6484b6-f7d6-4308-9cec-5631de746ee1"). InnerVolumeSpecName "kube-api-access-b42dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:28:20.462006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.461959 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b42dd\" (UniqueName: \"kubernetes.io/projected/6c6484b6-f7d6-4308-9cec-5631de746ee1-kube-api-access-b42dd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:28:20.462006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.462003 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6484b6-f7d6-4308-9cec-5631de746ee1-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:28:20.721495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.720794 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:28:20.743495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.727626 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 15:28:20.743495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.727706 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 15:28:20.969830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.969790 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" event={"ID":"6c6484b6-f7d6-4308-9cec-5631de746ee1","Type":"ContainerDied","Data":"31f1931f2a8369f46466d6a265db6f02d25105cb4207a283d66d0310885fe407"} Oct 31 15:28:20.970321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.969841 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31f1931f2a8369f46466d6a265db6f02d25105cb4207a283d66d0310885fe407" Oct 31 15:28:20.970321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:20.969889 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln" Oct 31 15:28:21.045178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:21.043942 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 15:28:21.045178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:21.044015 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 15:28:21.711496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:21.708800 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 15:28:21.711496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:21.708872 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 15:28:22.044565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:22.043315 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 15:28:22.044565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:22.043376 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 15:28:22.702493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:22.701751 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 15:28:22.702493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:22.701809 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 15:28:23.043990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:23.043943 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 15:28:23.044198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:23.044016 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 15:28:23.349539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:23.348937 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:23.643622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:23.642294 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:23.724129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:23.724091 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 15:28:23.724461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:23.724429 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 15:28:23.767594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:23.767560 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:24.021103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.021066 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:24.043993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.043955 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 15:28:24.044325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.044281 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 15:28:24.456782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.456672 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:24.714488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.711304 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body= Oct 31 15:28:24.714488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.711363 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" Oct 31 15:28:24.776704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.776663 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:28:24.976494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.976406 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw"] Oct 31 15:28:24.977083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.977070 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6c6484b6-f7d6-4308-9cec-5631de746ee1" containerName="revoke-keys" Oct 31 15:28:24.977146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.977084 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6484b6-f7d6-4308-9cec-5631de746ee1" containerName="revoke-keys" Oct 31 15:28:24.977193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.977184 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6484b6-f7d6-4308-9cec-5631de746ee1" containerName="revoke-keys" Oct 31 15:28:24.980016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:24.979997 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.034518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.034488 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-55b3fb-openshift-default-dockercfg-m7snf\"" Oct 31 15:28:25.044078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.044045 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 15:28:25.044368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.044342 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 15:28:25.073149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073114 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89f0e4-3499-4802-9a05-72062f42def8-istiod-ca-cert\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073163 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-socket\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073249 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89f0e4-3499-4802-9a05-72062f42def8-istio-podinfo\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073393 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-envoy\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073531 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctg28\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-kube-api-access-ctg28\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073626 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-credential-socket\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073736 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-data\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073797 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-certs\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.073874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.073854 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-istio-token\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.091287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.090970 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:25.091653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.091632 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw"] Oct 31 15:28:25.221496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.178346 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89f0e4-3499-4802-9a05-72062f42def8-istiod-ca-cert\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.237572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.182336 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89f0e4-3499-4802-9a05-72062f42def8-istiod-ca-cert\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.237820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.237784 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-socket\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.238063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.238046 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89f0e4-3499-4802-9a05-72062f42def8-istio-podinfo\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.242534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.238216 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-envoy\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.250533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.242809 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctg28\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-kube-api-access-ctg28\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.250803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.238844 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-socket\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.251433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.250951 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-credential-socket\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.271899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.271867 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-data\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.272281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.272248 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-certs\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.273151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.273130 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-istio-token\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.273526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.251374 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-credential-socket\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.273678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.273072 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-data\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.277960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.257767 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-envoy\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.278141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.272796 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-certs\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.290904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.290856 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:25.311163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.310061 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89f0e4-3499-4802-9a05-72062f42def8-istio-podinfo\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.322542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.320338 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctg28\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-kube-api-access-ctg28\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.346514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.345075 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-istio-token\") pod \"e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.593174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.593086 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:25.707742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.707714 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:28:25.709138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:25.709115 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 15:28:26.023942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.023908 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw"] Oct 31 15:28:26.026625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.026593 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:26.026755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.026647 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:26.026755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.026677 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:26.047457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.047435 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:28:26.047906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.047886 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:28:26.048723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.048705 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:28:26.055175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:26.055153 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" event={"ID":"cf89f0e4-3499-4802-9a05-72062f42def8","Type":"ContainerStarted","Data":"5a60d974bdc5da0db40657c00b59a261e209a74709b5521adbfe762723dbe9bc"} Oct 31 15:28:27.064004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:27.063969 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" event={"ID":"cf89f0e4-3499-4802-9a05-72062f42def8","Type":"ContainerStarted","Data":"2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754"} Oct 31 15:28:27.285452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:27.285399 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" podStartSLOduration=3.285386381 podStartE2EDuration="3.285386381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:27.232225337 +0000 UTC m=+2552.851776747" watchObservedRunningTime="2026-10-31 15:28:27.285386381 +0000 UTC m=+2552.904937792" Oct 31 15:28:27.594058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:27.594029 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:27.599610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:27.599588 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:28.071330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:28.071296 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:28.072334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:28.072315 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:28:28.394939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:28.394856 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:28.549598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:28.549561 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:28.685539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:28.685424 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:29.047722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:29.047690 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7"] Oct 31 15:28:29.051730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:29.051709 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e1d077-pjpk7"] Oct 31 15:28:30.017070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:30.017015 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:30.240412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:30.240381 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:30.372110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:30.372032 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:30.525886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:30.525840 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f001d4-d0d3-47d0-971a-b03b3157f1fd" path="/var/lib/kubelet/pods/38f001d4-d0d3-47d0-971a-b03b3157f1fd/volumes" Oct 31 15:28:31.051178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:31.051143 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr"] Oct 31 15:28:31.102052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:31.102021 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036612-8vghr"] Oct 31 15:28:32.498031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:32.497998 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2238581-8e2d-4216-aecb-b2d9baa08799" path="/var/lib/kubelet/pods/f2238581-8e2d-4216-aecb-b2d9baa08799/volumes" Oct 31 15:28:33.291321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.291255 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:33.472495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.469531 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:33.751790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.751742 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:33.885494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.881149 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh"] Oct 31 15:28:33.894059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.894032 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:33.944582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.944538 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh"] Oct 31 15:28:33.981359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.981319 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-55b3fb\"" Oct 31 15:28:33.981627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:33.981447 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-55b3fb-dockercfg-gfxkb\"" Oct 31 15:28:34.015513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.015409 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e424c501-ac9c-400a-90df-1062892d7e7f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:34.015652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.015627 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsffv\" (UniqueName: \"kubernetes.io/projected/e424c501-ac9c-400a-90df-1062892d7e7f-kube-api-access-wsffv\") pod \"payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:34.019792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.019773 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv"] Oct 31 15:28:34.022847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.022832 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.078591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.078567 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv"] Oct 31 15:28:34.117091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.117066 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11b4071-920e-4101-99c4-f4809253ae82-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.117245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.117120 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8gw\" (UniqueName: \"kubernetes.io/projected/a11b4071-920e-4101-99c4-f4809253ae82-kube-api-access-ql8gw\") pod \"payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.117319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.117302 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e424c501-ac9c-400a-90df-1062892d7e7f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:34.117462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.117446 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsffv\" (UniqueName: \"kubernetes.io/projected/e424c501-ac9c-400a-90df-1062892d7e7f-kube-api-access-wsffv\") pod \"payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:34.118101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.118071 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e424c501-ac9c-400a-90df-1062892d7e7f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:34.178800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.178771 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsffv\" (UniqueName: \"kubernetes.io/projected/e424c501-ac9c-400a-90df-1062892d7e7f-kube-api-access-wsffv\") pod \"payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:34.219126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.219095 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11b4071-920e-4101-99c4-f4809253ae82-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.219295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.219153 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8gw\" (UniqueName: \"kubernetes.io/projected/a11b4071-920e-4101-99c4-f4809253ae82-kube-api-access-ql8gw\") pod \"payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.219823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.219800 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11b4071-920e-4101-99c4-f4809253ae82-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.239667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.239643 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:34.277270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.277192 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8gw\" (UniqueName: \"kubernetes.io/projected/a11b4071-920e-4101-99c4-f4809253ae82-kube-api-access-ql8gw\") pod \"payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.335770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.335743 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:34.619057 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:28:34.619024 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode424c501_ac9c_400a_90df_1062892d7e7f.slice/crio-e9701b34be885faa3f53f7b6756cd4e779ea49f9735a3be4ed3923afa6c482a5 WatchSource:0}: Error finding container e9701b34be885faa3f53f7b6756cd4e779ea49f9735a3be4ed3923afa6c482a5: Status 404 returned error can't find the container with id e9701b34be885faa3f53f7b6756cd4e779ea49f9735a3be4ed3923afa6c482a5 Oct 31 15:28:34.620409 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.620385 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh"] Oct 31 15:28:34.727985 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:28:34.727946 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11b4071_920e_4101_99c4_f4809253ae82.slice/crio-8a4c3dde8e5614eb74700cb11157776d9eab47abce93f3e6d118c4d1b3ba13d5 WatchSource:0}: Error finding container 8a4c3dde8e5614eb74700cb11157776d9eab47abce93f3e6d118c4d1b3ba13d5: Status 404 returned error can't find the container with id 8a4c3dde8e5614eb74700cb11157776d9eab47abce93f3e6d118c4d1b3ba13d5 Oct 31 15:28:34.728721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:34.728700 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv"] Oct 31 15:28:35.132820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.132778 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" event={"ID":"e424c501-ac9c-400a-90df-1062892d7e7f","Type":"ContainerStarted","Data":"7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2"} Oct 31 15:28:35.133329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.132844 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" event={"ID":"e424c501-ac9c-400a-90df-1062892d7e7f","Type":"ContainerStarted","Data":"e9701b34be885faa3f53f7b6756cd4e779ea49f9735a3be4ed3923afa6c482a5"} Oct 31 15:28:35.133329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.132875 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:35.134491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.134453 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" event={"ID":"a11b4071-920e-4101-99c4-f4809253ae82","Type":"ContainerStarted","Data":"ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e"} Oct 31 15:28:35.134610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.134512 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" event={"ID":"a11b4071-920e-4101-99c4-f4809253ae82","Type":"ContainerStarted","Data":"8a4c3dde8e5614eb74700cb11157776d9eab47abce93f3e6d118c4d1b3ba13d5"} Oct 31 15:28:35.134610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.134541 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:35.250229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.250067 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" podStartSLOduration=2.250049867 podStartE2EDuration="2.250049867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:35.24669352 +0000 UTC m=+2560.866244960" watchObservedRunningTime="2026-10-31 15:28:35.250049867 +0000 UTC m=+2560.869601279" Oct 31 15:28:35.364783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:35.364721 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" podStartSLOduration=2.364699067 podStartE2EDuration="2.364699067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:35.362339772 +0000 UTC m=+2560.981891185" watchObservedRunningTime="2026-10-31 15:28:35.364699067 +0000 UTC m=+2560.984250478" Oct 31 15:28:38.699109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:38.699064 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:43.701687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:43.701649 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:44.089107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:44.089072 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:44.526634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:44.526601 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:45.061005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.060976 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv"] Oct 31 15:28:45.064364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.064347 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.117644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.117603 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-aa864d-openshift-default-dockercfg-rchsq\"" Oct 31 15:28:45.169351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169318 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56cg\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-kube-api-access-v56cg\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169398 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169563 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-socket\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169629 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-certs\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169763 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-podinfo\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169797 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-data\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169840 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-token\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169918 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-credential-socket\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.169988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.169952 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-envoy\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.173988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.173968 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv"] Oct 31 15:28:45.201249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.201222 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:45.309777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.278735 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-socket\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.326587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.281809 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-socket\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.326901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.326876 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-certs\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.327525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.327501 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-podinfo\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.353205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.353151 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-data\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.353428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.353212 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-token\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.353428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.353287 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-credential-socket\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.353428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.353327 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-envoy\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.353649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.353505 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56cg\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-kube-api-access-v56cg\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.353649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.353586 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.354222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.327321 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-certs\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.354658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.354635 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istiod-ca-cert\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.355250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.355223 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-data\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.355829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.355805 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-credential-socket\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.362761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.358025 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:45.362761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.360929 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-podinfo\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.363448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.363426 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-envoy\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.413829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.413796 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-token\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.415461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.415435 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56cg\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-kube-api-access-v56cg\") pod \"e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:45.678018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:45.677926 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:46.050753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.050718 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q"] Oct 31 15:28:46.054898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.054876 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2392e8d0-whp7q"] Oct 31 15:28:46.144660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.144633 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 15:28:46.145101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.144694 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 15:28:46.161262 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:28:46.161231 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d447b6_4d3d_41dd_8a2f_ba4e13cf46c1.slice/crio-31f954805b1d9003adda44c4bde7e4b0980c79002936a796dd71651b84664c22 WatchSource:0}: Error finding container 31f954805b1d9003adda44c4bde7e4b0980c79002936a796dd71651b84664c22: Status 404 returned error can't find the container with id 31f954805b1d9003adda44c4bde7e4b0980c79002936a796dd71651b84664c22 Oct 31 15:28:46.163585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.163564 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv"] Oct 31 15:28:46.163995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.163966 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:46.164039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.164028 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:46.164075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.164056 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:46.363586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.363502 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" event={"ID":"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1","Type":"ContainerStarted","Data":"bd51cd0fecb5c65f1f7c455ff55ebffc7c12d2e280c649e8be8ed8704c7bac1c"} Oct 31 15:28:46.363586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.363570 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" event={"ID":"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1","Type":"ContainerStarted","Data":"31f954805b1d9003adda44c4bde7e4b0980c79002936a796dd71651b84664c22"} Oct 31 15:28:46.495691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.495636 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" podStartSLOduration=1.495624591 podStartE2EDuration="1.495624591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:46.494983051 +0000 UTC m=+2572.114534463" watchObservedRunningTime="2026-10-31 15:28:46.495624591 +0000 UTC m=+2572.115176001" Oct 31 15:28:46.500800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.500770 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19a4567-d5b8-4699-a337-4470bbd810b4" path="/var/lib/kubelet/pods/c19a4567-d5b8-4699-a337-4470bbd810b4/volumes" Oct 31 15:28:46.678776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:46.678686 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:47.490524 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:28:47.490488 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:28:47.685517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:47.685421 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:48.050167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:48.050134 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj"] Oct 31 15:28:48.054189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:48.054167 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c645eee1-2jnbj"] Oct 31 15:28:48.381654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:48.381569 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:48.382574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:48.382557 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:28:48.498009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:48.497977 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ec01b3-76a8-48af-bd5f-1b6d065704f4" path="/var/lib/kubelet/pods/26ec01b3-76a8-48af-bd5f-1b6d065704f4/volumes" Oct 31 15:28:48.521145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:48.521112 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:49.862616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:49.862582 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:50.056511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:50.056452 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:50.213667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:50.213636 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:50.354834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:50.354803 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:51.258263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.258224 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9"] Oct 31 15:28:51.262698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.262672 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.341094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.341060 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-46bf36-openshift-default-dockercfg-zkvfl\"" Oct 31 15:28:51.346799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.346776 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.346872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.346823 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-podinfo\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.346872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.346849 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-token\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.346999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.346981 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bglpj\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-kube-api-access-bglpj\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.347049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.347035 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-socket\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.347137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.347117 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-envoy\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.347202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.347189 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-credential-socket\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.347275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.347264 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-data\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.347340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.347329 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-certs\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.385708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.385678 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9"] Oct 31 15:28:51.434861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.434825 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:51.449016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.448987 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-podinfo\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449022 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-token\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449094 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bglpj\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-kube-api-access-bglpj\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449141 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-socket\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449174 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-envoy\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449226 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-credential-socket\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449329 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-data\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449381 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-certs\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449444 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449705 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-socket\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449724 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-data\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.449849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.449797 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-credential-socket\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.450121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.450097 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-certs\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.450405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.450382 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istiod-ca-cert\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.451575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.451549 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-envoy\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.451889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.451866 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-podinfo\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.522593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.522515 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bglpj\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-kube-api-access-bglpj\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.523308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.523258 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-token\") pod \"e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:51.571148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.571117 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:51.578684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:51.578662 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:52.016384 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:28:52.016348 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4a93c2_2dc1_4b9a_8cf0_821630eea00d.slice/crio-d44514fdc013e463a267d92749f0fea4416b3c5b9a3b2dbd705e0544387e344f WatchSource:0}: Error finding container d44514fdc013e463a267d92749f0fea4416b3c5b9a3b2dbd705e0544387e344f: Status 404 returned error can't find the container with id d44514fdc013e463a267d92749f0fea4416b3c5b9a3b2dbd705e0544387e344f Oct 31 15:28:52.018454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.018436 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9"] Oct 31 15:28:52.019426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.019395 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:52.019515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.019500 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:52.019558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.019543 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:28:52.417732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.417643 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" event={"ID":"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d","Type":"ContainerStarted","Data":"f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593"} Oct 31 15:28:52.417732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.417696 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" event={"ID":"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d","Type":"ContainerStarted","Data":"d44514fdc013e463a267d92749f0fea4416b3c5b9a3b2dbd705e0544387e344f"} Oct 31 15:28:52.538774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.538724 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podStartSLOduration=1.538710103 podStartE2EDuration="1.538710103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:52.535608278 +0000 UTC m=+2578.155159692" watchObservedRunningTime="2026-10-31 15:28:52.538710103 +0000 UTC m=+2578.158261513" Oct 31 15:28:52.579126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:52.579095 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:53.584932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:53.584901 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:53.684817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:53.684751 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:53.912825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:53.912663 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:54.105573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.105492 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:54.323370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.323342 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d"] Oct 31 15:28:54.327938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.327922 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.381268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.381242 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-aa864d-dockercfg-54nms\"" Oct 31 15:28:54.381413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.381315 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-aa864d\"" Oct 31 15:28:54.385713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.385689 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d"] Oct 31 15:28:54.494791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.494744 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:54.530247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.530104 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f1e816-7f06-4c01-9d69-eb51b87787e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.530857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.530817 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5qq\" (UniqueName: \"kubernetes.io/projected/52f1e816-7f06-4c01-9d69-eb51b87787e3-kube-api-access-gq5qq\") pod \"payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.552738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.552708 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:28:54.557399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.556172 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt"] Oct 31 15:28:54.564076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.562956 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt"] Oct 31 15:28:54.564076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.563063 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:54.632122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.632089 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4swd\" (UniqueName: \"kubernetes.io/projected/6951eea9-0a8d-4278-88e0-982426951c8c-kube-api-access-n4swd\") pod \"payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:54.632612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.632176 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5qq\" (UniqueName: \"kubernetes.io/projected/52f1e816-7f06-4c01-9d69-eb51b87787e3-kube-api-access-gq5qq\") pod \"payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.632612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.632263 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6951eea9-0a8d-4278-88e0-982426951c8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:54.632612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.632302 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f1e816-7f06-4c01-9d69-eb51b87787e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.633004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.632987 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f1e816-7f06-4c01-9d69-eb51b87787e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.690089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.690062 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5qq\" (UniqueName: \"kubernetes.io/projected/52f1e816-7f06-4c01-9d69-eb51b87787e3-kube-api-access-gq5qq\") pod \"payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.733483 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.733431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4swd\" (UniqueName: \"kubernetes.io/projected/6951eea9-0a8d-4278-88e0-982426951c8c-kube-api-access-n4swd\") pod \"payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:54.733660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.733591 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6951eea9-0a8d-4278-88e0-982426951c8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:54.734436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.734408 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6951eea9-0a8d-4278-88e0-982426951c8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:54.791457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.791424 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4swd\" (UniqueName: \"kubernetes.io/projected/6951eea9-0a8d-4278-88e0-982426951c8c-kube-api-access-n4swd\") pod \"payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:54.939798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.939720 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-aa864d-dockercfg-54nms\"" Oct 31 15:28:54.942993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.942970 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:54.948115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:54.948100 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:55.325708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.325678 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d"] Oct 31 15:28:55.362446 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:28:55.362414 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f1e816_7f06_4c01_9d69_eb51b87787e3.slice/crio-f2edc69870b89c1d1911e6f6af6508e9f3018c9804df1711e726c3d0ab7f51b7 WatchSource:0}: Error finding container f2edc69870b89c1d1911e6f6af6508e9f3018c9804df1711e726c3d0ab7f51b7: Status 404 returned error can't find the container with id f2edc69870b89c1d1911e6f6af6508e9f3018c9804df1711e726c3d0ab7f51b7 Oct 31 15:28:55.374807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.374784 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt"] Oct 31 15:28:55.376746 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:28:55.376715 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6951eea9_0a8d_4278_88e0_982426951c8c.slice/crio-28b0f03df3ecfadc5afef9e5c025fff23b6bebe85cab6ae18734cc744c63d8e7 WatchSource:0}: Error finding container 28b0f03df3ecfadc5afef9e5c025fff23b6bebe85cab6ae18734cc744c63d8e7: Status 404 returned error can't find the container with id 28b0f03df3ecfadc5afef9e5c025fff23b6bebe85cab6ae18734cc744c63d8e7 Oct 31 15:28:55.478569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.478539 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:55.504201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.504163 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" event={"ID":"6951eea9-0a8d-4278-88e0-982426951c8c","Type":"ContainerStarted","Data":"6f055f551c0688ec0bbbb2e35b25278ee2b626059e35e092dc373b4a1746ec21"} Oct 31 15:28:55.504826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.504235 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" event={"ID":"6951eea9-0a8d-4278-88e0-982426951c8c","Type":"ContainerStarted","Data":"28b0f03df3ecfadc5afef9e5c025fff23b6bebe85cab6ae18734cc744c63d8e7"} Oct 31 15:28:55.504826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.504268 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:28:55.506643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.506604 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" event={"ID":"52f1e816-7f06-4c01-9d69-eb51b87787e3","Type":"ContainerStarted","Data":"96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2"} Oct 31 15:28:55.506752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.506656 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" event={"ID":"52f1e816-7f06-4c01-9d69-eb51b87787e3","Type":"ContainerStarted","Data":"f2edc69870b89c1d1911e6f6af6508e9f3018c9804df1711e726c3d0ab7f51b7"} Oct 31 15:28:55.506752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.506681 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:28:55.626719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.626605 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" podStartSLOduration=1.626588195 podStartE2EDuration="1.626588195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:55.62189864 +0000 UTC m=+2581.241450051" watchObservedRunningTime="2026-10-31 15:28:55.626588195 +0000 UTC m=+2581.246139607" Oct 31 15:28:55.657384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.657356 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:55.767491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.766195 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" podStartSLOduration=1.7661715070000001 podStartE2EDuration="1.766171507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:55.739647251 +0000 UTC m=+2581.359198662" watchObservedRunningTime="2026-10-31 15:28:55.766171507 +0000 UTC m=+2581.385722919" Oct 31 15:28:55.799519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:55.799453 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:58.047716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:58.047686 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx"] Oct 31 15:28:58.098354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:58.098324 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d542de-8l4nx"] Oct 31 15:28:58.245687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:58.245657 4865 scope.go:122] "RemoveContainer" containerID="51ac34c4b8c3f45df60a78cacd1eff49b33fa6960ae5feb4d11795c53ee760f1" Oct 31 15:28:58.498401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:58.498365 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b04c111-d276-4acb-81a3-a3ada6b44fe9" path="/var/lib/kubelet/pods/5b04c111-d276-4acb-81a3-a3ada6b44fe9/volumes" Oct 31 15:28:58.698730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:58.698699 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:58.875160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:58.875087 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:59.014492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.013883 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:59.102423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.102388 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7"] Oct 31 15:28:59.108225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.108208 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.216883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.216853 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-46bf36-dockercfg-4dwpn\"" Oct 31 15:28:59.232539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.230778 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6xb\" (UniqueName: \"kubernetes.io/projected/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-kube-api-access-qw6xb\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.255657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.232893 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.289561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.231406 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-46bf36\"" Oct 31 15:28:59.350512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.350445 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7"] Oct 31 15:28:59.358251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.358206 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6xb\" (UniqueName: \"kubernetes.io/projected/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-kube-api-access-qw6xb\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.358421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.358375 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.359784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.359723 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.374041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.374008 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl"] Oct 31 15:28:59.379878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.379855 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl"] Oct 31 15:28:59.380006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.379954 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.416814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.416793 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6xb\" (UniqueName: \"kubernetes.io/projected/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-kube-api-access-qw6xb\") pod \"payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.423654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.423631 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:28:59.460122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.460086 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e274d0a9-6d82-4faa-b2d7-85e3d882c094-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.460329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.460300 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5bn\" (UniqueName: \"kubernetes.io/projected/e274d0a9-6d82-4faa-b2d7-85e3d882c094-kube-api-access-5x5bn\") pod \"payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.561142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.561112 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5bn\" (UniqueName: \"kubernetes.io/projected/e274d0a9-6d82-4faa-b2d7-85e3d882c094-kube-api-access-5x5bn\") pod \"payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.561358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.561292 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e274d0a9-6d82-4faa-b2d7-85e3d882c094-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.562027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.562003 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e274d0a9-6d82-4faa-b2d7-85e3d882c094-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.618844 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.618809 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5bn\" (UniqueName: \"kubernetes.io/projected/e274d0a9-6d82-4faa-b2d7-85e3d882c094-kube-api-access-5x5bn\") pod \"payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.628074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.628029 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:28:59.696244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.696216 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:28:59.808950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:28:59.808203 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7"] Oct 31 15:29:00.080277 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:29:00.080237 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode274d0a9_6d82_4faa_b2d7_85e3d882c094.slice/crio-2fdd8c6d437609f02da43e741f307a53b202706cbe082b88e2950e1c3093ff3f WatchSource:0}: Error finding container 2fdd8c6d437609f02da43e741f307a53b202706cbe082b88e2950e1c3093ff3f: Status 404 returned error can't find the container with id 2fdd8c6d437609f02da43e741f307a53b202706cbe082b88e2950e1c3093ff3f Oct 31 15:29:00.081202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.081184 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl"] Oct 31 15:29:00.551692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.551655 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" event={"ID":"e274d0a9-6d82-4faa-b2d7-85e3d882c094","Type":"ContainerStarted","Data":"70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612"} Oct 31 15:29:00.552115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.551720 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" event={"ID":"e274d0a9-6d82-4faa-b2d7-85e3d882c094","Type":"ContainerStarted","Data":"2fdd8c6d437609f02da43e741f307a53b202706cbe082b88e2950e1c3093ff3f"} Oct 31 15:29:00.552115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.551843 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:29:00.553406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.553382 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" event={"ID":"e76f37ab-69f6-4d07-8da5-ad04ce5d334b","Type":"ContainerStarted","Data":"db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0"} Oct 31 15:29:00.553538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.553434 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:29:00.553538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.553451 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" event={"ID":"e76f37ab-69f6-4d07-8da5-ad04ce5d334b","Type":"ContainerStarted","Data":"80a901f4c5eb27698f4bf8d9fc1966dc003cb4232587df71a12f3aa82778dde2"} Oct 31 15:29:00.668085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.668031 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" podStartSLOduration=1.668016856 podStartE2EDuration="1.668016856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:00.666374966 +0000 UTC m=+2586.285926388" watchObservedRunningTime="2026-10-31 15:29:00.668016856 +0000 UTC m=+2586.287568267" Oct 31 15:29:00.782827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.782455 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" podStartSLOduration=1.782437118 podStartE2EDuration="1.782437118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:00.781782787 +0000 UTC m=+2586.401334199" watchObservedRunningTime="2026-10-31 15:29:00.782437118 +0000 UTC m=+2586.401988532" Oct 31 15:29:00.921790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:00.921101 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:01.954654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:01.954624 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:29:03.919783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:03.919748 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:05.048830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:05.048797 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn"] Oct 31 15:29:05.122870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:05.122841 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576b9ee-j7qhn"] Oct 31 15:29:06.520485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:06.520428 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df64a25d-11c1-4e7e-a956-bffd0ebf0a13" path="/var/lib/kubelet/pods/df64a25d-11c1-4e7e-a956-bffd0ebf0a13/volumes" Oct 31 15:29:06.521640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:06.521611 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 15:29:06.522126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:06.522098 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 15:29:08.778244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:08.778207 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:11.564600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:11.564570 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:29:11.565097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:11.564636 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:29:12.675126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:12.675032 4865 generic.go:356] "Generic (PLEG): container finished" podID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerID="b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f" exitCode=0 Oct 31 15:29:12.675126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:12.675105 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" event={"ID":"eb908f7b-125c-4283-a4cd-09e603a2c4d9","Type":"ContainerDied","Data":"b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f"} Oct 31 15:29:12.675678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:12.675573 4865 scope.go:122] "RemoveContainer" containerID="b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f" Oct 31 15:29:12.677375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:12.677348 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:29:12.677504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:12.677419 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:29:12.677504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:12.677455 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:29:13.442493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:13.442451 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:13.686531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:13.686498 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" event={"ID":"eb908f7b-125c-4283-a4cd-09e603a2c4d9","Type":"ContainerStarted","Data":"b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c"} Oct 31 15:29:14.050154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:14.050117 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4"] Oct 31 15:29:14.054517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:14.054494 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6ba2273-z8rz4"] Oct 31 15:29:14.382333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:14.382250 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:29:14.382530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:14.382422 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 15:29:14.382530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:14.382496 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 15:29:14.526078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:14.526028 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a43aa0f-3730-4d71-b414-e44ae053f8ee" path="/var/lib/kubelet/pods/4a43aa0f-3730-4d71-b414-e44ae053f8ee/volumes" Oct 31 15:29:15.383388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:15.383349 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 15:29:15.383845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:15.383420 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 15:29:16.382770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:16.382737 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 15:29:16.383037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:16.382787 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 15:29:17.382497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:17.382427 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 15:29:17.382929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:17.382526 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 15:29:17.526079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:17.526044 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:17.531901 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:29:17.531863 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:29:18.390339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:18.389864 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 15:29:18.390339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:18.389948 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 15:29:19.183840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.183809 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:19.309885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.309841 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:19.387913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.387807 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:29:19.388269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.388248 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:29:19.389662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.389630 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:29:19.404599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.404564 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln"] Oct 31 15:29:19.454991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.454913 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d9cddd-mxrln"] Oct 31 15:29:19.620023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.619987 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr"] Oct 31 15:29:19.620239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.620213 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" containerID="cri-o://50b6509da1c676de1791f947bba50902ccc1423ff35d0d1443bf8cd7186daf2f" gracePeriod=30 Oct 31 15:29:19.629247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.629220 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6"] Oct 31 15:29:19.629640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.629612 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" containerID="cri-o://a8f9f1365cb3ed30eee6958db0f467dd27348d75ea2ad4cd24934c194fc29b7a" gracePeriod=30 Oct 31 15:29:19.660288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.660253 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:19.707445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.707413 4865 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:29:19.707647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:19.707491 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:29:20.383086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:20.383055 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:20.497955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:20.497924 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6484b6-f7d6-4308-9cec-5631de746ee1" path="/var/lib/kubelet/pods/6c6484b6-f7d6-4308-9cec-5631de746ee1/volumes" Oct 31 15:29:21.088005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:21.087971 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:21.217832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:21.217778 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:22.050017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:22.049965 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8"] Oct 31 15:29:22.100907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:22.100875 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d270a049-fpsv8"] Oct 31 15:29:22.497771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:22.497738 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7af2eb-aa55-4f08-b71f-6023f92018eb" path="/var/lib/kubelet/pods/4d7af2eb-aa55-4f08-b71f-6023f92018eb/volumes" Oct 31 15:29:24.047963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.047928 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8"] Oct 31 15:29:24.098359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.098334 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d26e41-6mbg8"] Oct 31 15:29:24.379529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.379443 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:24.499600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.499565 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4bfdd6-a901-4bdc-9214-37c101e83dde" path="/var/lib/kubelet/pods/ba4bfdd6-a901-4bdc-9214-37c101e83dde/volumes" Oct 31 15:29:24.615631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.615598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:24.743690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.743660 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:24.823814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.823785 4865 generic.go:356] "Generic (PLEG): container finished" podID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerID="50b6509da1c676de1791f947bba50902ccc1423ff35d0d1443bf8cd7186daf2f" exitCode=0 Oct 31 15:29:24.824036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.824015 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" event={"ID":"2e3d7b5b-3430-4c12-9a55-58b5723f4c95","Type":"ContainerDied","Data":"50b6509da1c676de1791f947bba50902ccc1423ff35d0d1443bf8cd7186daf2f"} Oct 31 15:29:24.828316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.828016 4865 generic.go:356] "Generic (PLEG): container finished" podID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerID="a8f9f1365cb3ed30eee6958db0f467dd27348d75ea2ad4cd24934c194fc29b7a" exitCode=0 Oct 31 15:29:24.828316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.828148 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" event={"ID":"14345f00-2d2a-4c48-b57f-93e63037a10d","Type":"ContainerDied","Data":"a8f9f1365cb3ed30eee6958db0f467dd27348d75ea2ad4cd24934c194fc29b7a"} Oct 31 15:29:24.828316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.828198 4865 scope.go:122] "RemoveContainer" containerID="4e740e2a31bd440e8ec13ae5040fd96b43a8c5bef6ff96d047b945beba15ae6b" Oct 31 15:29:24.909627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.909603 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:29:24.928817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.928798 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:29:24.931124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931101 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-credential-socket\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931189 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-certs\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931264 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-envoy\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931359 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-credential-socket" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:24.931457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931366 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-socket\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931454 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-certs" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:24.931606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931546 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-podinfo\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931588 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-socket" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:24.931718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931609 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-token\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931658 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istiod-ca-cert\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931710 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-data\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.931871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.931738 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-kube-api-access-bscwd\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-kube-api-access-bscwd\") pod \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\" (UID: \"2e3d7b5b-3430-4c12-9a55-58b5723f4c95\") " Oct 31 15:29:24.932127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.932030 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istiod-ca-cert" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:24.932537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.932516 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-data" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:24.932647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.932581 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:24.932647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.932599 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:24.932647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.932609 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:24.932647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.932618 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:24.933773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.933757 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-podinfo" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:29:24.933850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.933787 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-envoy" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:24.934338 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.934318 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-kube-api-access-bscwd" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "kube-api-access-bscwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:24.934403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:24.934351 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-token" pod "2e3d7b5b-3430-4c12-9a55-58b5723f4c95" (UID: "2e3d7b5b-3430-4c12-9a55-58b5723f4c95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:25.033642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.033608 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-data\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.033803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.033752 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-socket\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.033803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.033792 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14345f00-2d2a-4c48-b57f-93e63037a10d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14345f00-2d2a-4c48-b57f-93e63037a10d-istiod-ca-cert\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.033912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.033849 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-credential-socket\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.033912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.033903 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-envoy\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.034018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.033921 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-data" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.034018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.033932 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-certs\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.034132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.034047 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-token\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-token\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.034197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.034157 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-kube-api-access-bt6gm\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-kube-api-access-bt6gm\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.034263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.034203 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-certs" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.034263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.034244 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-credential-socket" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.034370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.034262 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14345f00-2d2a-4c48-b57f-93e63037a10d-istiod-ca-cert" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:25.034428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.034362 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-podinfo\") pod \"14345f00-2d2a-4c48-b57f-93e63037a10d\" (UID: \"14345f00-2d2a-4c48-b57f-93e63037a10d\") " Oct 31 15:29:25.034894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.034858 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-socket" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035248 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035276 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscwd\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-kube-api-access-bscwd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035292 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035307 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035321 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035335 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14345f00-2d2a-4c48-b57f-93e63037a10d-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035349 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035363 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3d7b5b-3430-4c12-9a55-58b5723f4c95-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035377 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.035420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.035392 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.036584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.036562 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-token" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:25.036729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.036713 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-kube-api-access-bt6gm" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "kube-api-access-bt6gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:25.037021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.037000 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-podinfo" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:29:25.037270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.037254 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-envoy" pod "14345f00-2d2a-4c48-b57f-93e63037a10d" (UID: "14345f00-2d2a-4c48-b57f-93e63037a10d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.136907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.136833 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.136907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.136857 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.136907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.136867 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6gm\" (UniqueName: \"kubernetes.io/projected/14345f00-2d2a-4c48-b57f-93e63037a10d-kube-api-access-bt6gm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.136907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.136875 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14345f00-2d2a-4c48-b57f-93e63037a10d-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.842125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.842091 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" event={"ID":"2e3d7b5b-3430-4c12-9a55-58b5723f4c95","Type":"ContainerDied","Data":"42d638a05b2c1f45c7b29a3a0b8d923dd33f8c0b2b82eef290015d75feb0a476"} Oct 31 15:29:25.842323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.842157 4865 scope.go:122] "RemoveContainer" containerID="50b6509da1c676de1791f947bba50902ccc1423ff35d0d1443bf8cd7186daf2f" Oct 31 15:29:25.842323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.842101 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr" Oct 31 15:29:25.843862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.843842 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" event={"ID":"14345f00-2d2a-4c48-b57f-93e63037a10d","Type":"ContainerDied","Data":"f90d99ae79fc7b87a8f60d59e0c9ca7ff274a2277bfd2c1cb1c9484a9fad8123"} Oct 31 15:29:25.843960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.843850 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6" Oct 31 15:29:25.854035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:25.854018 4865 scope.go:122] "RemoveContainer" containerID="a8f9f1365cb3ed30eee6958db0f467dd27348d75ea2ad4cd24934c194fc29b7a" Oct 31 15:29:26.014611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.014581 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr"] Oct 31 15:29:26.065514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.065488 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-7ggtr"] Oct 31 15:29:26.177524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.177439 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6"] Oct 31 15:29:26.181485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.181449 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-f74m6"] Oct 31 15:29:26.195703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.195679 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz"] Oct 31 15:29:26.306497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.306103 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247"] Oct 31 15:29:26.450381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.450352 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:26.500246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.500213 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" path="/var/lib/kubelet/pods/14345f00-2d2a-4c48-b57f-93e63037a10d/volumes" Oct 31 15:29:26.500814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.500792 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" path="/var/lib/kubelet/pods/2e3d7b5b-3430-4c12-9a55-58b5723f4c95/volumes" Oct 31 15:29:26.574177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.574081 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:26.697379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:26.697347 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:29.807044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:29.807014 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:31.196388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.196351 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" podUID="0fd1fe3f-3174-400f-b20c-ab3fca64ed88" containerName="payload-processing" containerID="cri-o://917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253" gracePeriod=25 Oct 31 15:29:31.307119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.307080 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" podUID="a6329fe9-80df-4026-8628-ccc4468eb8e1" containerName="payload-pre-processing" containerID="cri-o://d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7" gracePeriod=25 Oct 31 15:29:31.476636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.476610 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:29:31.538734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.538701 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-kube-api-access-djwjc\" (UniqueName: \"kubernetes.io/projected/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-kube-api-access-djwjc\") pod \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " Oct 31 15:29:31.538923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.538834 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-plugins-config-volume\") pod \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\" (UID: \"0fd1fe3f-3174-400f-b20c-ab3fca64ed88\") " Oct 31 15:29:31.539307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.539277 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-plugins-config-volume" pod "0fd1fe3f-3174-400f-b20c-ab3fca64ed88" (UID: "0fd1fe3f-3174-400f-b20c-ab3fca64ed88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:31.543254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.543224 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-kube-api-access-djwjc" pod "0fd1fe3f-3174-400f-b20c-ab3fca64ed88" (UID: "0fd1fe3f-3174-400f-b20c-ab3fca64ed88"). InnerVolumeSpecName "kube-api-access-djwjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:31.587416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.587393 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:29:31.641082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.641035 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djwjc\" (UniqueName: \"kubernetes.io/projected/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-kube-api-access-djwjc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:31.641082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.641061 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd1fe3f-3174-400f-b20c-ab3fca64ed88-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:31.742174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.742143 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6329fe9-80df-4026-8628-ccc4468eb8e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6329fe9-80df-4026-8628-ccc4468eb8e1-plugins-config-volume\") pod \"a6329fe9-80df-4026-8628-ccc4468eb8e1\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " Oct 31 15:29:31.742376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.742244 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6329fe9-80df-4026-8628-ccc4468eb8e1-kube-api-access-z522d\" (UniqueName: \"kubernetes.io/projected/a6329fe9-80df-4026-8628-ccc4468eb8e1-kube-api-access-z522d\") pod \"a6329fe9-80df-4026-8628-ccc4468eb8e1\" (UID: \"a6329fe9-80df-4026-8628-ccc4468eb8e1\") " Oct 31 15:29:31.742584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.742563 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6329fe9-80df-4026-8628-ccc4468eb8e1-plugins-config-volume" pod "a6329fe9-80df-4026-8628-ccc4468eb8e1" (UID: "a6329fe9-80df-4026-8628-ccc4468eb8e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:31.742886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.742868 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6329fe9-80df-4026-8628-ccc4468eb8e1-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:31.744499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.744443 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6329fe9-80df-4026-8628-ccc4468eb8e1-kube-api-access-z522d" pod "a6329fe9-80df-4026-8628-ccc4468eb8e1" (UID: "a6329fe9-80df-4026-8628-ccc4468eb8e1"). InnerVolumeSpecName "kube-api-access-z522d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:31.844592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.844566 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z522d\" (UniqueName: \"kubernetes.io/projected/a6329fe9-80df-4026-8628-ccc4468eb8e1-kube-api-access-z522d\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:31.895793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.895735 4865 generic.go:356] "Generic (PLEG): container finished" podID="0fd1fe3f-3174-400f-b20c-ab3fca64ed88" containerID="917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253" exitCode=0 Oct 31 15:29:31.895793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.895778 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" event={"ID":"0fd1fe3f-3174-400f-b20c-ab3fca64ed88","Type":"ContainerDied","Data":"917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253"} Oct 31 15:29:31.896006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.895797 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" Oct 31 15:29:31.896006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.895872 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz" event={"ID":"0fd1fe3f-3174-400f-b20c-ab3fca64ed88","Type":"ContainerDied","Data":"808c255024d4937e262cc92281a750e615e2e458cabf15b02f305b5a97f06054"} Oct 31 15:29:31.896006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.895904 4865 scope.go:122] "RemoveContainer" containerID="917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253" Oct 31 15:29:31.897757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.897735 4865 generic.go:356] "Generic (PLEG): container finished" podID="a6329fe9-80df-4026-8628-ccc4468eb8e1" containerID="d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7" exitCode=0 Oct 31 15:29:31.897884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.897770 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" event={"ID":"a6329fe9-80df-4026-8628-ccc4468eb8e1","Type":"ContainerDied","Data":"d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7"} Oct 31 15:29:31.897884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.897794 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" Oct 31 15:29:31.897884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.897831 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247" event={"ID":"a6329fe9-80df-4026-8628-ccc4468eb8e1","Type":"ContainerDied","Data":"4c3678501fb481908a00c4719433a53d3ea31715e6c48027ad715fc0a6d7aac2"} Oct 31 15:29:31.907414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.907391 4865 scope.go:122] "RemoveContainer" containerID="917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253" Oct 31 15:29:31.907752 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:29:31.907730 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253\": container with ID starting with 917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253 not found: ID does not exist" containerID="917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253" Oct 31 15:29:31.907840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.907759 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253"} err="failed to get container status \"917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253\": rpc error: code = NotFound desc = could not find container \"917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253\": container with ID starting with 917fc9382edb7f40c1c4f34b424728bdff85c2b115ffe03a98859e9947d48253 not found: ID does not exist" Oct 31 15:29:31.907840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.907777 4865 scope.go:122] "RemoveContainer" containerID="d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7" Oct 31 15:29:31.917880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.917863 4865 scope.go:122] "RemoveContainer" containerID="d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7" Oct 31 15:29:31.918158 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:29:31.918139 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7\": container with ID starting with d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7 not found: ID does not exist" containerID="d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7" Oct 31 15:29:31.918232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.918167 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7"} err="failed to get container status \"d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7\": rpc error: code = NotFound desc = could not find container \"d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7\": container with ID starting with d95edc856a25dbfef2a33b6bab5c6f25d8d9efd1fdaf579e2a23a7bdf116b3d7 not found: ID does not exist" Oct 31 15:29:31.945515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:31.945465 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:32.063612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:32.063541 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz"] Oct 31 15:29:32.069636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:32.069611 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebadea-566c785957-bk2nz"] Oct 31 15:29:32.181281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:32.181249 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247"] Oct 31 15:29:32.185156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:32.185135 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebadea-7bdbff7d75-2k247"] Oct 31 15:29:32.497255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:32.497223 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd1fe3f-3174-400f-b20c-ab3fca64ed88" path="/var/lib/kubelet/pods/0fd1fe3f-3174-400f-b20c-ab3fca64ed88/volumes" Oct 31 15:29:32.497734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:32.497718 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6329fe9-80df-4026-8628-ccc4468eb8e1" path="/var/lib/kubelet/pods/a6329fe9-80df-4026-8628-ccc4468eb8e1/volumes" Oct 31 15:29:34.584794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:34.584762 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:36.263091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:36.263055 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:37.346604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.346570 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:37.855820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.855788 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx"] Oct 31 15:29:37.856417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856405 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0fd1fe3f-3174-400f-b20c-ab3fca64ed88" containerName="payload-processing" Oct 31 15:29:37.856464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856419 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd1fe3f-3174-400f-b20c-ab3fca64ed88" containerName="payload-processing" Oct 31 15:29:37.856464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856437 4865 cpu_manager.go:414] "containerMap: removing container" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" Oct 31 15:29:37.856464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856443 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" Oct 31 15:29:37.856464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856453 4865 cpu_manager.go:414] "containerMap: removing container" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" Oct 31 15:29:37.856464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856460 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" Oct 31 15:29:37.856464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856486 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a6329fe9-80df-4026-8628-ccc4468eb8e1" containerName="payload-pre-processing" Oct 31 15:29:37.856464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856492 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6329fe9-80df-4026-8628-ccc4468eb8e1" containerName="payload-pre-processing" Oct 31 15:29:37.856726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856519 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" Oct 31 15:29:37.856726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856524 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" Oct 31 15:29:37.856726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856614 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" Oct 31 15:29:37.856726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856621 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6329fe9-80df-4026-8628-ccc4468eb8e1" containerName="payload-pre-processing" Oct 31 15:29:37.856726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856633 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3d7b5b-3430-4c12-9a55-58b5723f4c95" containerName="istio-proxy" Oct 31 15:29:37.856726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.856642 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd1fe3f-3174-400f-b20c-ab3fca64ed88" containerName="payload-processing" Oct 31 15:29:37.862038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:37.862017 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.039642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.039601 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b25761-openshift-default-dockercfg-5zqmh\"" Oct 31 15:29:38.045972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.045934 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:38.051807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.051618 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx"] Oct 31 15:29:38.150852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.150763 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-certs\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.150852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.150802 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1195e70e-0cf4-433c-884d-9c99833cef37-istio-podinfo\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.150852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.150828 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.151108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.151042 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-credential-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.151170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.151117 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-data\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.151279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.151260 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1195e70e-0cf4-433c-884d-9c99833cef37-istiod-ca-cert\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.151344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.151308 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-istio-token\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.151414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.151397 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5b8\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-kube-api-access-ld5b8\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.151504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.151485 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-envoy\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.252547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252514 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-envoy\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.252756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252594 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-certs\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.252756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252615 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1195e70e-0cf4-433c-884d-9c99833cef37-istio-podinfo\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.252756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252638 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.252756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252690 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-credential-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.252756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252727 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-data\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252812 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1195e70e-0cf4-433c-884d-9c99833cef37-istiod-ca-cert\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252847 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-istio-token\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.252878 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5b8\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-kube-api-access-ld5b8\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.253039 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-certs\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.253103 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-credential-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.253147 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.253336 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-data\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.253564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.253550 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1195e70e-0cf4-433c-884d-9c99833cef37-istiod-ca-cert\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.255020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.254999 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-envoy\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.255292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.255276 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1195e70e-0cf4-433c-884d-9c99833cef37-istio-podinfo\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.316974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.316946 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-istio-token\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.317532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.317512 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5b8\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-kube-api-access-ld5b8\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.475920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.475890 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:38.495282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.495250 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:38.907571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.907485 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx"] Oct 31 15:29:38.945761 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:29:38.945732 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1195e70e_0cf4_433c_884d_9c99833cef37.slice/crio-c7ba3b4c14b0e8dca63a38958223a0d2590ea60f6a7b9e6985261e84faa05c90 WatchSource:0}: Error finding container c7ba3b4c14b0e8dca63a38958223a0d2590ea60f6a7b9e6985261e84faa05c90: Status 404 returned error can't find the container with id c7ba3b4c14b0e8dca63a38958223a0d2590ea60f6a7b9e6985261e84faa05c90 Oct 31 15:29:38.948418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.948383 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:29:38.948533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.948451 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:29:38.948533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:38.948527 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:29:39.029554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:39.029514 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:39.071679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:39.071634 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" event={"ID":"1195e70e-0cf4-433c-884d-9c99833cef37","Type":"ContainerStarted","Data":"66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf"} Oct 31 15:29:39.071952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:39.071699 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" event={"ID":"1195e70e-0cf4-433c-884d-9c99833cef37","Type":"ContainerStarted","Data":"c7ba3b4c14b0e8dca63a38958223a0d2590ea60f6a7b9e6985261e84faa05c90"} Oct 31 15:29:39.193285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:39.193150 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podStartSLOduration=2.193134287 podStartE2EDuration="2.193134287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:39.190795285 +0000 UTC m=+2624.810346725" watchObservedRunningTime="2026-10-31 15:29:39.193134287 +0000 UTC m=+2624.812685740" Oct 31 15:29:39.476193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:39.476167 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:40.481954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:40.481923 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:40.909537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:40.909422 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:41.088350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:41.088310 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:41.089290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:41.089271 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:29:41.990761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:41.990732 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:42.048285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:42.048254 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb"] Oct 31 15:29:42.055011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:42.054992 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ca8bf32-6ltzb"] Oct 31 15:29:42.154933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:42.154905 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:42.286153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:42.286077 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:42.498107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:42.498078 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a22e53-2e40-4024-a602-12f0c16918a0" path="/var/lib/kubelet/pods/92a22e53-2e40-4024-a602-12f0c16918a0/volumes" Oct 31 15:29:45.151973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:45.151942 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:45.327451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:45.325056 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:46.010609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:46.010571 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:46.052539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:46.052513 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f"] Oct 31 15:29:46.103021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:46.102994 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b788651-h7j9f"] Oct 31 15:29:46.204575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:46.204545 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:46.334887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:46.334800 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:46.467983 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:46.467955 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:46.497442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:46.497414 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74dd8be-2cad-4699-b645-a4f905e5feca" path="/var/lib/kubelet/pods/f74dd8be-2cad-4699-b645-a4f905e5feca/volumes" Oct 31 15:29:47.491273 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:29:47.491238 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:29:47.888305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:47.888268 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:48.364247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:48.364215 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:48.386534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:48.386505 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk"] Oct 31 15:29:48.386791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:48.386766 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" containerID="cri-o://7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565" gracePeriod=30 Oct 31 15:29:51.233098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:51.233042 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:51.366004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:51.365917 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:53.658728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.658669 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:29:53.819140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819109 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-credential-socket\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819203 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/489b6427-e570-48d5-8ded-d0065ffc6393-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489b6427-e570-48d5-8ded-d0065ffc6393-istio-podinfo\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819236 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-certs\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819277 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-kube-api-access-dghl4\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-kube-api-access-dghl4\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819382 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-envoy\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819434 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-socket\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819464 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-credential-socket" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:53.819625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819525 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-istio-token\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-istio-token\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819583 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-certs" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:53.819790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819696 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-data\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819735 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489b6427-e570-48d5-8ded-d0065ffc6393-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489b6427-e570-48d5-8ded-d0065ffc6393-istiod-ca-cert\") pod \"489b6427-e570-48d5-8ded-d0065ffc6393\" (UID: \"489b6427-e570-48d5-8ded-d0065ffc6393\") " Oct 31 15:29:53.819905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819843 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-socket" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:53.820051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.819997 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-data" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:53.820244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.820214 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489b6427-e570-48d5-8ded-d0065ffc6393-istiod-ca-cert" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:53.820806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.820777 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.820922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.820811 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.820922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.820827 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.820922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.820840 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.820922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.820856 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/489b6427-e570-48d5-8ded-d0065ffc6393-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.821956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.821933 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-envoy" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:53.822055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.821950 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-kube-api-access-dghl4" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "kube-api-access-dghl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:53.822162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.822136 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/489b6427-e570-48d5-8ded-d0065ffc6393-istio-podinfo" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:29:53.822277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.822190 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-istio-token" pod "489b6427-e570-48d5-8ded-d0065ffc6393" (UID: "489b6427-e570-48d5-8ded-d0065ffc6393"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:53.922018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.921930 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/489b6427-e570-48d5-8ded-d0065ffc6393-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.922018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.921960 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghl4\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-kube-api-access-dghl4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.922018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.921986 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/489b6427-e570-48d5-8ded-d0065ffc6393-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:53.922018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:53.921995 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/489b6427-e570-48d5-8ded-d0065ffc6393-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:54.197001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.196914 4865 generic.go:356] "Generic (PLEG): container finished" podID="489b6427-e570-48d5-8ded-d0065ffc6393" containerID="7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565" exitCode=0 Oct 31 15:29:54.197001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.196957 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" event={"ID":"489b6427-e570-48d5-8ded-d0065ffc6393","Type":"ContainerDied","Data":"7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565"} Oct 31 15:29:54.197285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.197030 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" event={"ID":"489b6427-e570-48d5-8ded-d0065ffc6393","Type":"ContainerDied","Data":"5ea67681c0029b0d32c3d2e066fffe884abc3b063def33ae38736e709733b615"} Oct 31 15:29:54.197285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.196966 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk" Oct 31 15:29:54.197285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.197063 4865 scope.go:122] "RemoveContainer" containerID="7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565" Oct 31 15:29:54.208985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.208962 4865 scope.go:122] "RemoveContainer" containerID="82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01" Oct 31 15:29:54.219439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.219420 4865 scope.go:122] "RemoveContainer" containerID="7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565" Oct 31 15:29:54.219743 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:29:54.219724 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565\": container with ID starting with 7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565 not found: ID does not exist" containerID="7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565" Oct 31 15:29:54.219829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.219755 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565"} err="failed to get container status \"7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565\": rpc error: code = NotFound desc = could not find container \"7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565\": container with ID starting with 7a2aa779d8158ea91cc7b3281b58548e1fc92f6cfb9508ad4ca33bf4cd55c565 not found: ID does not exist" Oct 31 15:29:54.219829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.219780 4865 scope.go:122] "RemoveContainer" containerID="82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01" Oct 31 15:29:54.220051 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:29:54.220030 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01\": container with ID starting with 82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01 not found: ID does not exist" containerID="82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01" Oct 31 15:29:54.220093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.220059 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01"} err="failed to get container status \"82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01\": rpc error: code = NotFound desc = could not find container \"82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01\": container with ID starting with 82f44ba47b7d2336b0cdca26b6c64f4e6934719701acdd1c025c1731502f9a01 not found: ID does not exist" Oct 31 15:29:54.368141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.368112 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk"] Oct 31 15:29:54.418712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.418679 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebadea-openshift-default-7d75c49869-8vhkk"] Oct 31 15:29:54.497909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:54.497877 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" path="/var/lib/kubelet/pods/489b6427-e570-48d5-8ded-d0065ffc6393/volumes" Oct 31 15:29:55.936800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:55.936756 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:56.095552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:56.095518 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:56.357497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:56.357419 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:29:58.281712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:58.281685 4865 scope.go:122] "RemoveContainer" containerID="2db14fdcc71d160aea5baa7e04f1adc455d55122292f90e214da1a7b892b9d16" Oct 31 15:29:58.292043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:58.292026 4865 scope.go:122] "RemoveContainer" containerID="3e801405cf71af0328a0c3868dc5d7d5529d84d4cfce3f253836065ba94a71ed" Oct 31 15:29:58.303444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:58.303424 4865 scope.go:122] "RemoveContainer" containerID="196a19ca79ba034b2a56e5171b12b4cfeca166522610e0e8a67acadcf522e33d" Oct 31 15:29:58.312736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:58.312714 4865 scope.go:122] "RemoveContainer" containerID="30baee89c62657024672402bed286d673be04f567769ea5ca80295bab90371e1" Oct 31 15:29:58.324817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:58.324794 4865 scope.go:122] "RemoveContainer" containerID="463aa41e18f02fcb1c9066401f26f2ce7fd6c61acdb28773cb531c12ff7e47ac" Oct 31 15:29:58.333453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:58.333437 4865 scope.go:122] "RemoveContainer" containerID="e9560d4163646968f844b4e318af791191d207b708035a51c95697f355c0b23f" Oct 31 15:29:59.978160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:29:59.978089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:00.163078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.163045 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn"] Oct 31 15:30:00.163812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.163789 4865 cpu_manager.go:414] "containerMap: removing container" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" Oct 31 15:30:00.163812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.163813 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" Oct 31 15:30:00.164006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.163870 4865 cpu_manager.go:414] "containerMap: removing container" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" Oct 31 15:30:00.164006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.163879 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" Oct 31 15:30:00.164117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.164048 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" Oct 31 15:30:00.164117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.164066 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="14345f00-2d2a-4c48-b57f-93e63037a10d" containerName="istio-proxy" Oct 31 15:30:00.169748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.169727 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj"] Oct 31 15:30:00.169885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.169839 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.170548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.170526 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="489b6427-e570-48d5-8ded-d0065ffc6393" containerName="istio-proxy" Oct 31 15:30:00.174057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.174039 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn"] Oct 31 15:30:00.174190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.174140 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.178173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.178153 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.213071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.213044 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg"] Oct 31 15:30:00.217236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.217220 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr"] Oct 31 15:30:00.217366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.217346 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.221327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221309 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t"] Oct 31 15:30:00.221439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221375 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45ae376-b982-439f-8d90-10b0e90974ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.221439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221412 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8zv5\" (UniqueName: \"kubernetes.io/projected/040913dd-2664-4379-a9c3-c77b61e9c5b0-kube-api-access-d8zv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.221439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221421 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.221705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221677 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cdc270-e7df-4e32-b486-78063e4d3de5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.221775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221758 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfm7c\" (UniqueName: \"kubernetes.io/projected/69cdc270-e7df-4e32-b486-78063e4d3de5-kube-api-access-rfm7c\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.221840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221813 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/040913dd-2664-4379-a9c3-c77b61e9c5b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.221935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.221919 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xn4n\" (UniqueName: \"kubernetes.io/projected/c45ae376-b982-439f-8d90-10b0e90974ef-kube-api-access-7xn4n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.225656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.225445 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw"] Oct 31 15:30:00.225656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.225569 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.230306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.230259 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2"] Oct 31 15:30:00.230405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.230375 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.234166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.234148 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4"] Oct 31 15:30:00.234302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.234281 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.238343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.238322 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.260886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.260861 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:30:00.260992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.260867 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:30:00.260992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.260913 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:30:00.265548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.265524 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg"] Oct 31 15:30:00.269746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.269728 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj"] Oct 31 15:30:00.273814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.273797 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn"] Oct 31 15:30:00.277038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.277021 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr"] Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.319538 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t"] Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.325567 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw"] Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.325577 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqz4m\" (UniqueName: \"kubernetes.io/projected/76d82b74-7865-4772-be6c-38b4d7a2660b-kube-api-access-fqz4m\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.325666 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef53331-8c08-41e5-81be-70ed9d941e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.325714 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cdc270-e7df-4e32-b486-78063e4d3de5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.325820 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtp6\" (UniqueName: \"kubernetes.io/projected/bef53331-8c08-41e5-81be-70ed9d941e4b-kube-api-access-wgtp6\") pod \"maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.325849 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5nc5\" (UniqueName: \"kubernetes.io/projected/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-kube-api-access-p5nc5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.325941 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621c588b-3dc6-40ae-a1c2-a6ddb214669b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326009 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfm7c\" (UniqueName: \"kubernetes.io/projected/69cdc270-e7df-4e32-b486-78063e4d3de5-kube-api-access-rfm7c\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326061 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326088 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b21624-f0ba-4179-827b-7dfb79f51d39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326117 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/040913dd-2664-4379-a9c3-c77b61e9c5b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326156 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d82b74-7865-4772-be6c-38b4d7a2660b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326203 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgp72\" (UniqueName: \"kubernetes.io/projected/c16ecf29-3bde-4477-85bc-59e0291549fa-kube-api-access-lgp72\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326260 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16ecf29-3bde-4477-85bc-59e0291549fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326325 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xn4n\" (UniqueName: \"kubernetes.io/projected/c45ae376-b982-439f-8d90-10b0e90974ef-kube-api-access-7xn4n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326385 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45ae376-b982-439f-8d90-10b0e90974ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326412 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrsqh\" (UniqueName: \"kubernetes.io/projected/60b21624-f0ba-4179-827b-7dfb79f51d39-kube-api-access-mrsqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326439 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zv5\" (UniqueName: \"kubernetes.io/projected/040913dd-2664-4379-a9c3-c77b61e9c5b0-kube-api-access-d8zv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.326520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.326516 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cdc270-e7df-4e32-b486-78063e4d3de5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.327871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.327214 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/040913dd-2664-4379-a9c3-c77b61e9c5b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.327871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.327270 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrr6\" (UniqueName: \"kubernetes.io/projected/621c588b-3dc6-40ae-a1c2-a6ddb214669b-kube-api-access-rhrr6\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.327871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.327459 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45ae376-b982-439f-8d90-10b0e90974ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.333189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.333166 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2"] Oct 31 15:30:00.369155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.369090 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4"] Oct 31 15:30:00.373975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.373949 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn"] Oct 31 15:30:00.385322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.385297 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zv5\" (UniqueName: \"kubernetes.io/projected/040913dd-2664-4379-a9c3-c77b61e9c5b0-kube-api-access-d8zv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.385444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.385302 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xn4n\" (UniqueName: \"kubernetes.io/projected/c45ae376-b982-439f-8d90-10b0e90974ef-kube-api-access-7xn4n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.386061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.386040 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfm7c\" (UniqueName: \"kubernetes.io/projected/69cdc270-e7df-4e32-b486-78063e4d3de5-kube-api-access-rfm7c\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.429205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429158 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtp6\" (UniqueName: \"kubernetes.io/projected/bef53331-8c08-41e5-81be-70ed9d941e4b-kube-api-access-wgtp6\") pod \"maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.429364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429212 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5nc5\" (UniqueName: \"kubernetes.io/projected/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-kube-api-access-p5nc5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.429364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429246 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621c588b-3dc6-40ae-a1c2-a6ddb214669b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.429364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429331 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.429364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429358 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b21624-f0ba-4179-827b-7dfb79f51d39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.429624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429504 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d82b74-7865-4772-be6c-38b4d7a2660b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.429624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429554 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgp72\" (UniqueName: \"kubernetes.io/projected/c16ecf29-3bde-4477-85bc-59e0291549fa-kube-api-access-lgp72\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.429624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429591 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16ecf29-3bde-4477-85bc-59e0291549fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.429802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429659 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrsqh\" (UniqueName: \"kubernetes.io/projected/60b21624-f0ba-4179-827b-7dfb79f51d39-kube-api-access-mrsqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.429802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429737 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrr6\" (UniqueName: \"kubernetes.io/projected/621c588b-3dc6-40ae-a1c2-a6ddb214669b-kube-api-access-rhrr6\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.429802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429782 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqz4m\" (UniqueName: \"kubernetes.io/projected/76d82b74-7865-4772-be6c-38b4d7a2660b-kube-api-access-fqz4m\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.429969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.429875 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef53331-8c08-41e5-81be-70ed9d941e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.430230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.430202 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b21624-f0ba-4179-827b-7dfb79f51d39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.430304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.430232 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d82b74-7865-4772-be6c-38b4d7a2660b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.430304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.430199 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.430304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.430250 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621c588b-3dc6-40ae-a1c2-a6ddb214669b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.430452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.430428 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16ecf29-3bde-4477-85bc-59e0291549fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.430604 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.430587 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef53331-8c08-41e5-81be-70ed9d941e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.485279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.485199 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:00.488149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.488119 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5nc5\" (UniqueName: \"kubernetes.io/projected/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-kube-api-access-p5nc5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.488381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.488362 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrr6\" (UniqueName: \"kubernetes.io/projected/621c588b-3dc6-40ae-a1c2-a6ddb214669b-kube-api-access-rhrr6\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.488501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.488434 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrsqh\" (UniqueName: \"kubernetes.io/projected/60b21624-f0ba-4179-827b-7dfb79f51d39-kube-api-access-mrsqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.488700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.488677 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgp72\" (UniqueName: \"kubernetes.io/projected/c16ecf29-3bde-4477-85bc-59e0291549fa-kube-api-access-lgp72\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.489396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.488887 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqz4m\" (UniqueName: \"kubernetes.io/projected/76d82b74-7865-4772-be6c-38b4d7a2660b-kube-api-access-fqz4m\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.489525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.489368 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtp6\" (UniqueName: \"kubernetes.io/projected/bef53331-8c08-41e5-81be-70ed9d941e4b-kube-api-access-wgtp6\") pod \"maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.493303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.493274 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:00.500795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.500773 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:00.534805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.534732 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:00.542837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.542652 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:00.550953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.550845 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:00.558375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.558345 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:00.574042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.571221 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:00.574042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.573863 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:00.817071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.816983 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:00.897523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:00.897233 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn"] Oct 31 15:30:01.175557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.152265 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f"] Oct 31 15:30:01.175557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.152586 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr"] Oct 31 15:30:01.175557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.163196 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg"] Oct 31 15:30:01.175557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.170499 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512a45ea-hzn2f"] Oct 31 15:30:01.178804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.178730 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj"] Oct 31 15:30:01.189906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.182648 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn"] Oct 31 15:30:01.189906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.188481 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4"] Oct 31 15:30:01.198284 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:30:01.198260 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cdc270_e7df_4e32_b486_78063e4d3de5.slice/crio-f67e112424e44f90fb06381959f0039770ae39f9b30f0908b2cf74c29ce5d0ec WatchSource:0}: Error finding container f67e112424e44f90fb06381959f0039770ae39f9b30f0908b2cf74c29ce5d0ec: Status 404 returned error can't find the container with id f67e112424e44f90fb06381959f0039770ae39f9b30f0908b2cf74c29ce5d0ec Oct 31 15:30:01.242268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.242244 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:30:01.252716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.252688 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerStarted","Data":"80cc6d2a9ed6d3eaa2bf6b26cf65f1349eb843c4ac0693a91a3920983997b0f5"} Oct 31 15:30:01.254027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.253992 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" event={"ID":"bef53331-8c08-41e5-81be-70ed9d941e4b","Type":"ContainerStarted","Data":"a407987607842e7116f7ba5d5a82d7c742a37340fb95d57ef1bf3e62fa9ec77d"} Oct 31 15:30:01.255243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.255219 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" event={"ID":"c45ae376-b982-439f-8d90-10b0e90974ef","Type":"ContainerStarted","Data":"1de9925405cb664fe2be22d58865aa5bf0b32c8bb4f14dc86287713b7502715b"} Oct 31 15:30:01.256552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.256523 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" event={"ID":"69cdc270-e7df-4e32-b486-78063e4d3de5","Type":"ContainerStarted","Data":"f67e112424e44f90fb06381959f0039770ae39f9b30f0908b2cf74c29ce5d0ec"} Oct 31 15:30:01.258087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.258063 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" event={"ID":"040913dd-2664-4379-a9c3-c77b61e9c5b0","Type":"ContainerStarted","Data":"a39685a333946c39295d8f6ea53064ca583698283b3d50f76654d81fa1e5d32c"} Oct 31 15:30:01.259453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.259431 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerStarted","Data":"38ec249005c46efe20f7b52e9a30d173134ce24fb24899138ec949b2d4ec1b30"} Oct 31 15:30:01.284799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.284775 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw"] Oct 31 15:30:01.296524 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:30:01.296457 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b21624_f0ba_4179_827b_7dfb79f51d39.slice/crio-e88f0c8b829345012cb904d3e20b9a20feef534021cb9863a282d58cc980665e WatchSource:0}: Error finding container e88f0c8b829345012cb904d3e20b9a20feef534021cb9863a282d58cc980665e: Status 404 returned error can't find the container with id e88f0c8b829345012cb904d3e20b9a20feef534021cb9863a282d58cc980665e Oct 31 15:30:01.310637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.310613 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2"] Oct 31 15:30:01.311065 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:30:01.311037 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d82b74_7865_4772_be6c_38b4d7a2660b.slice/crio-f4751bed284bb69d5d24b507d02debe48e8bf229e4148b003d5243a0b672c2f6 WatchSource:0}: Error finding container f4751bed284bb69d5d24b507d02debe48e8bf229e4148b003d5243a0b672c2f6: Status 404 returned error can't find the container with id f4751bed284bb69d5d24b507d02debe48e8bf229e4148b003d5243a0b672c2f6 Oct 31 15:30:01.380445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.380418 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:01.390091 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:30:01.390069 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16ecf29_3bde_4477_85bc_59e0291549fa.slice/crio-1b5ffc479470becd6dbd81ec1178bf288b9caad03c3aea78cae67d14eb9c7fc4 WatchSource:0}: Error finding container 1b5ffc479470becd6dbd81ec1178bf288b9caad03c3aea78cae67d14eb9c7fc4: Status 404 returned error can't find the container with id 1b5ffc479470becd6dbd81ec1178bf288b9caad03c3aea78cae67d14eb9c7fc4 Oct 31 15:30:01.391319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:01.391297 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t"] Oct 31 15:30:02.280093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.279990 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" event={"ID":"c16ecf29-3bde-4477-85bc-59e0291549fa","Type":"ContainerStarted","Data":"8cfb6c5487ae1d9bec648b1df6b7a2d7eaa8aa88a23617caf64c9552f89c3438"} Oct 31 15:30:02.280093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.280090 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" event={"ID":"c16ecf29-3bde-4477-85bc-59e0291549fa","Type":"ContainerStarted","Data":"1b5ffc479470becd6dbd81ec1178bf288b9caad03c3aea78cae67d14eb9c7fc4"} Oct 31 15:30:02.282099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.282072 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerStarted","Data":"0480dd0902c319d0f2039ab6eeef46b6d0c131662cff72462328c1440488b66f"} Oct 31 15:30:02.283789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.283766 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" event={"ID":"60b21624-f0ba-4179-827b-7dfb79f51d39","Type":"ContainerStarted","Data":"cef8ea724d6976f91a4249ccf11d87b4d38742a5a3dd0e7db6cc4d2b51d06cf0"} Oct 31 15:30:02.283883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.283805 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" event={"ID":"60b21624-f0ba-4179-827b-7dfb79f51d39","Type":"ContainerStarted","Data":"e88f0c8b829345012cb904d3e20b9a20feef534021cb9863a282d58cc980665e"} Oct 31 15:30:02.285371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.285349 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" event={"ID":"bef53331-8c08-41e5-81be-70ed9d941e4b","Type":"ContainerStarted","Data":"fab7d0c945ee4a9f8f0b09c8eb34ef6f7ff1ca2baf903f5fecff02bb196eec66"} Oct 31 15:30:02.287078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.287056 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" event={"ID":"c45ae376-b982-439f-8d90-10b0e90974ef","Type":"ContainerStarted","Data":"ae06a6f67809945fa2a140f85b59353db57975f6f0457aba44dfcfb4b2160b11"} Oct 31 15:30:02.288594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.288569 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" event={"ID":"76d82b74-7865-4772-be6c-38b4d7a2660b","Type":"ContainerStarted","Data":"157b5cbfa8b8c3912ca37611b533303efa6380d41948ea62bac5feded8b82299"} Oct 31 15:30:02.288698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.288615 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" event={"ID":"76d82b74-7865-4772-be6c-38b4d7a2660b","Type":"ContainerStarted","Data":"f4751bed284bb69d5d24b507d02debe48e8bf229e4148b003d5243a0b672c2f6"} Oct 31 15:30:02.290102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.290078 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" event={"ID":"69cdc270-e7df-4e32-b486-78063e4d3de5","Type":"ContainerStarted","Data":"cfda5f561bb56347d8de3b1bb4aca95486945439efd3cccde62d214679dadcbc"} Oct 31 15:30:02.291682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.291655 4865 generic.go:356] "Generic (PLEG): container finished" podID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerID="071b5db66fa1a1a7a6d193f1fcf999af86efb57e6e3913f1f34f64d3b837af14" exitCode=60 Oct 31 15:30:02.291757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.291684 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" event={"ID":"040913dd-2664-4379-a9c3-c77b61e9c5b0","Type":"ContainerDied","Data":"071b5db66fa1a1a7a6d193f1fcf999af86efb57e6e3913f1f34f64d3b837af14"} Oct 31 15:30:02.291997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.291974 4865 scope.go:122] "RemoveContainer" containerID="071b5db66fa1a1a7a6d193f1fcf999af86efb57e6e3913f1f34f64d3b837af14" Oct 31 15:30:02.293351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.293328 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerStarted","Data":"3c02ca0764a07159a62aa33cde7d0a63dccf433fcce51dc8faace8974dcc3280"} Oct 31 15:30:02.399601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.397182 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" podStartSLOduration=2.397162991 podStartE2EDuration="2.397162991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:02.395976396 +0000 UTC m=+2648.015527824" watchObservedRunningTime="2026-10-31 15:30:02.397162991 +0000 UTC m=+2648.016714403" Oct 31 15:30:02.529542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.529483 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" podStartSLOduration=2.529445302 podStartE2EDuration="2.529445302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:02.526608547 +0000 UTC m=+2648.146159959" watchObservedRunningTime="2026-10-31 15:30:02.529445302 +0000 UTC m=+2648.148996714" Oct 31 15:30:02.533056 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.532983 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ff8b15-0993-48b7-aa54-137382b5c198" path="/var/lib/kubelet/pods/54ff8b15-0993-48b7-aa54-137382b5c198/volumes" Oct 31 15:30:02.833579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.817977 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" podStartSLOduration=2.817958504 podStartE2EDuration="2.817958504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:02.665530726 +0000 UTC m=+2648.285082135" watchObservedRunningTime="2026-10-31 15:30:02.817958504 +0000 UTC m=+2648.437509915" Oct 31 15:30:02.838068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:02.838009 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" podStartSLOduration=2.83798899 podStartE2EDuration="2.83798899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:02.825522686 +0000 UTC m=+2648.445074101" watchObservedRunningTime="2026-10-31 15:30:02.83798899 +0000 UTC m=+2648.457540402" Oct 31 15:30:03.085665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.085582 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" podStartSLOduration=3.085566035 podStartE2EDuration="3.085566035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:02.948295642 +0000 UTC m=+2648.567847054" watchObservedRunningTime="2026-10-31 15:30:03.085566035 +0000 UTC m=+2648.705117516" Oct 31 15:30:03.223495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.219709 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" podStartSLOduration=3.219689435 podStartE2EDuration="3.219689435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:03.216590479 +0000 UTC m=+2648.836141892" watchObservedRunningTime="2026-10-31 15:30:03.219689435 +0000 UTC m=+2648.839240847" Oct 31 15:30:03.351063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.350554 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" podStartSLOduration=3.350539153 podStartE2EDuration="3.350539153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:03.334131032 +0000 UTC m=+2648.953682443" watchObservedRunningTime="2026-10-31 15:30:03.350539153 +0000 UTC m=+2648.970090565" Oct 31 15:30:03.352459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.352431 4865 generic.go:356] "Generic (PLEG): container finished" podID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerID="8cfb6c5487ae1d9bec648b1df6b7a2d7eaa8aa88a23617caf64c9552f89c3438" exitCode=60 Oct 31 15:30:03.352793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.352603 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" event={"ID":"c16ecf29-3bde-4477-85bc-59e0291549fa","Type":"ContainerDied","Data":"8cfb6c5487ae1d9bec648b1df6b7a2d7eaa8aa88a23617caf64c9552f89c3438"} Oct 31 15:30:03.353309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.353014 4865 scope.go:122] "RemoveContainer" containerID="8cfb6c5487ae1d9bec648b1df6b7a2d7eaa8aa88a23617caf64c9552f89c3438" Oct 31 15:30:03.355594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.355570 4865 generic.go:356] "Generic (PLEG): container finished" podID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerID="0480dd0902c319d0f2039ab6eeef46b6d0c131662cff72462328c1440488b66f" exitCode=60 Oct 31 15:30:03.355972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.355954 4865 scope.go:122] "RemoveContainer" containerID="0480dd0902c319d0f2039ab6eeef46b6d0c131662cff72462328c1440488b66f" Oct 31 15:30:03.356039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.355996 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerDied","Data":"0480dd0902c319d0f2039ab6eeef46b6d0c131662cff72462328c1440488b66f"} Oct 31 15:30:03.358524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.358019 4865 generic.go:356] "Generic (PLEG): container finished" podID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerID="cef8ea724d6976f91a4249ccf11d87b4d38742a5a3dd0e7db6cc4d2b51d06cf0" exitCode=7 Oct 31 15:30:03.358524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.358317 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" event={"ID":"60b21624-f0ba-4179-827b-7dfb79f51d39","Type":"ContainerDied","Data":"cef8ea724d6976f91a4249ccf11d87b4d38742a5a3dd0e7db6cc4d2b51d06cf0"} Oct 31 15:30:03.358524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.358406 4865 scope.go:122] "RemoveContainer" containerID="cef8ea724d6976f91a4249ccf11d87b4d38742a5a3dd0e7db6cc4d2b51d06cf0" Oct 31 15:30:03.359932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.359906 4865 generic.go:356] "Generic (PLEG): container finished" podID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerID="fab7d0c945ee4a9f8f0b09c8eb34ef6f7ff1ca2baf903f5fecff02bb196eec66" exitCode=60 Oct 31 15:30:03.360035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.359989 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" event={"ID":"bef53331-8c08-41e5-81be-70ed9d941e4b","Type":"ContainerDied","Data":"fab7d0c945ee4a9f8f0b09c8eb34ef6f7ff1ca2baf903f5fecff02bb196eec66"} Oct 31 15:30:03.360298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.360280 4865 scope.go:122] "RemoveContainer" containerID="fab7d0c945ee4a9f8f0b09c8eb34ef6f7ff1ca2baf903f5fecff02bb196eec66" Oct 31 15:30:03.367751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.367730 4865 generic.go:356] "Generic (PLEG): container finished" podID="c45ae376-b982-439f-8d90-10b0e90974ef" containerID="ae06a6f67809945fa2a140f85b59353db57975f6f0457aba44dfcfb4b2160b11" exitCode=60 Oct 31 15:30:03.367842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.367826 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" event={"ID":"c45ae376-b982-439f-8d90-10b0e90974ef","Type":"ContainerDied","Data":"ae06a6f67809945fa2a140f85b59353db57975f6f0457aba44dfcfb4b2160b11"} Oct 31 15:30:03.368061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.368044 4865 scope.go:122] "RemoveContainer" containerID="ae06a6f67809945fa2a140f85b59353db57975f6f0457aba44dfcfb4b2160b11" Oct 31 15:30:03.371453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.371435 4865 generic.go:356] "Generic (PLEG): container finished" podID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerID="157b5cbfa8b8c3912ca37611b533303efa6380d41948ea62bac5feded8b82299" exitCode=60 Oct 31 15:30:03.371586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.371546 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" event={"ID":"76d82b74-7865-4772-be6c-38b4d7a2660b","Type":"ContainerDied","Data":"157b5cbfa8b8c3912ca37611b533303efa6380d41948ea62bac5feded8b82299"} Oct 31 15:30:03.371783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.371767 4865 scope.go:122] "RemoveContainer" containerID="157b5cbfa8b8c3912ca37611b533303efa6380d41948ea62bac5feded8b82299" Oct 31 15:30:03.373504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.373456 4865 generic.go:356] "Generic (PLEG): container finished" podID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerID="cfda5f561bb56347d8de3b1bb4aca95486945439efd3cccde62d214679dadcbc" exitCode=60 Oct 31 15:30:03.373605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.373506 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" event={"ID":"69cdc270-e7df-4e32-b486-78063e4d3de5","Type":"ContainerDied","Data":"cfda5f561bb56347d8de3b1bb4aca95486945439efd3cccde62d214679dadcbc"} Oct 31 15:30:03.373966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.373949 4865 scope.go:122] "RemoveContainer" containerID="cfda5f561bb56347d8de3b1bb4aca95486945439efd3cccde62d214679dadcbc" Oct 31 15:30:03.375809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.375789 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" event={"ID":"040913dd-2664-4379-a9c3-c77b61e9c5b0","Type":"ContainerStarted","Data":"6fb0e261474fa7dbbd3d728bd47080e6917b69b0ab7407c87c5e6de92ceace74"} Oct 31 15:30:03.384500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.383865 4865 generic.go:356] "Generic (PLEG): container finished" podID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerID="3c02ca0764a07159a62aa33cde7d0a63dccf433fcce51dc8faace8974dcc3280" exitCode=60 Oct 31 15:30:03.384500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.383904 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerDied","Data":"3c02ca0764a07159a62aa33cde7d0a63dccf433fcce51dc8faace8974dcc3280"} Oct 31 15:30:03.384500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.384206 4865 scope.go:122] "RemoveContainer" containerID="3c02ca0764a07159a62aa33cde7d0a63dccf433fcce51dc8faace8974dcc3280" Oct 31 15:30:03.449774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.449725 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" podStartSLOduration=3.449710434 podStartE2EDuration="3.449710434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:03.449030457 +0000 UTC m=+2649.068581869" watchObservedRunningTime="2026-10-31 15:30:03.449710434 +0000 UTC m=+2649.069261844" Oct 31 15:30:03.948547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:03.948319 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" podStartSLOduration=3.948299468 podStartE2EDuration="3.948299468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:03.945719237 +0000 UTC m=+2649.565270651" watchObservedRunningTime="2026-10-31 15:30:03.948299468 +0000 UTC m=+2649.567850881" Oct 31 15:30:04.423497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.420819 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" event={"ID":"c16ecf29-3bde-4477-85bc-59e0291549fa","Type":"ContainerStarted","Data":"2c3ccd80ae13885783d045e4eb762d3368ff7d310413e96a18303a997b00b3db"} Oct 31 15:30:04.441390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.441350 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerStarted","Data":"c868c42dbf0c3b4023e70490cceaf125362b638c1913eda67bdf7d7fad30cb7d"} Oct 31 15:30:04.444825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.444801 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" event={"ID":"60b21624-f0ba-4179-827b-7dfb79f51d39","Type":"ContainerStarted","Data":"acc420ab2c5f6fc8820785acc9c610758c2055f2b231dc3153b009a02e311abf"} Oct 31 15:30:04.453496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.452813 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" event={"ID":"bef53331-8c08-41e5-81be-70ed9d941e4b","Type":"ContainerStarted","Data":"d4d6a506bf27288a7ab5ff213ec8668776905d9ea0826218e3f04f4f4ea5be9e"} Oct 31 15:30:04.455425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.455404 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" event={"ID":"c45ae376-b982-439f-8d90-10b0e90974ef","Type":"ContainerStarted","Data":"e77cefae191a0ba68e0dcca852fc7eb0c716b968b788204ab3d279b66785239b"} Oct 31 15:30:04.460155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.460124 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" event={"ID":"76d82b74-7865-4772-be6c-38b4d7a2660b","Type":"ContainerStarted","Data":"bcb00b699585eea03335c9a70487af131ba78d6e6bf40838b2a28d90d1c8268d"} Oct 31 15:30:04.462404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.462381 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" event={"ID":"69cdc270-e7df-4e32-b486-78063e4d3de5","Type":"ContainerStarted","Data":"1b0622c2095c8f49236db7708b4e26ce324e563f695b9fe481b1a83474e2d81a"} Oct 31 15:30:04.464736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.464694 4865 generic.go:356] "Generic (PLEG): container finished" podID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerID="6fb0e261474fa7dbbd3d728bd47080e6917b69b0ab7407c87c5e6de92ceace74" exitCode=60 Oct 31 15:30:04.464855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.464810 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" event={"ID":"040913dd-2664-4379-a9c3-c77b61e9c5b0","Type":"ContainerDied","Data":"6fb0e261474fa7dbbd3d728bd47080e6917b69b0ab7407c87c5e6de92ceace74"} Oct 31 15:30:04.464855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.464852 4865 scope.go:122] "RemoveContainer" containerID="071b5db66fa1a1a7a6d193f1fcf999af86efb57e6e3913f1f34f64d3b837af14" Oct 31 15:30:04.464973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.464953 4865 scope.go:122] "RemoveContainer" containerID="6fb0e261474fa7dbbd3d728bd47080e6917b69b0ab7407c87c5e6de92ceace74" Oct 31 15:30:04.465191 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:04.465173 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn_odh-ai-gateway-infra(040913dd-2664-4379-a9c3-c77b61e9c5b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" Oct 31 15:30:04.466988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:04.466963 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerStarted","Data":"633545669c1c9abf4e0596ace220337bcec412dbdb651ddcfd8f2a5dbbf0e645"} Oct 31 15:30:05.476504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.476450 4865 generic.go:356] "Generic (PLEG): container finished" podID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerID="2c3ccd80ae13885783d045e4eb762d3368ff7d310413e96a18303a997b00b3db" exitCode=60 Oct 31 15:30:05.476942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.476531 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" event={"ID":"c16ecf29-3bde-4477-85bc-59e0291549fa","Type":"ContainerDied","Data":"2c3ccd80ae13885783d045e4eb762d3368ff7d310413e96a18303a997b00b3db"} Oct 31 15:30:05.476942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.476592 4865 scope.go:122] "RemoveContainer" containerID="8cfb6c5487ae1d9bec648b1df6b7a2d7eaa8aa88a23617caf64c9552f89c3438" Oct 31 15:30:05.477058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.476975 4865 scope.go:122] "RemoveContainer" containerID="2c3ccd80ae13885783d045e4eb762d3368ff7d310413e96a18303a997b00b3db" Oct 31 15:30:05.477659 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.477358 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t_odh-ai-gateway-infra(c16ecf29-3bde-4477-85bc-59e0291549fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" Oct 31 15:30:05.479419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.479396 4865 generic.go:356] "Generic (PLEG): container finished" podID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerID="c868c42dbf0c3b4023e70490cceaf125362b638c1913eda67bdf7d7fad30cb7d" exitCode=60 Oct 31 15:30:05.479527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.479491 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerDied","Data":"c868c42dbf0c3b4023e70490cceaf125362b638c1913eda67bdf7d7fad30cb7d"} Oct 31 15:30:05.479812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.479798 4865 scope.go:122] "RemoveContainer" containerID="c868c42dbf0c3b4023e70490cceaf125362b638c1913eda67bdf7d7fad30cb7d" Oct 31 15:30:05.480214 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.480195 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg_odh-ai-gateway-infra(621c588b-3dc6-40ae-a1c2-a6ddb214669b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" Oct 31 15:30:05.481335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.481308 4865 generic.go:356] "Generic (PLEG): container finished" podID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerID="acc420ab2c5f6fc8820785acc9c610758c2055f2b231dc3153b009a02e311abf" exitCode=7 Oct 31 15:30:05.481440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.481397 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" event={"ID":"60b21624-f0ba-4179-827b-7dfb79f51d39","Type":"ContainerDied","Data":"acc420ab2c5f6fc8820785acc9c610758c2055f2b231dc3153b009a02e311abf"} Oct 31 15:30:05.481965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.481674 4865 scope.go:122] "RemoveContainer" containerID="acc420ab2c5f6fc8820785acc9c610758c2055f2b231dc3153b009a02e311abf" Oct 31 15:30:05.481965 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.481905 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw_odh-ai-gateway-infra(60b21624-f0ba-4179-827b-7dfb79f51d39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" Oct 31 15:30:05.483893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.483864 4865 generic.go:356] "Generic (PLEG): container finished" podID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerID="d4d6a506bf27288a7ab5ff213ec8668776905d9ea0826218e3f04f4f4ea5be9e" exitCode=60 Oct 31 15:30:05.483987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.483948 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" event={"ID":"bef53331-8c08-41e5-81be-70ed9d941e4b","Type":"ContainerDied","Data":"d4d6a506bf27288a7ab5ff213ec8668776905d9ea0826218e3f04f4f4ea5be9e"} Oct 31 15:30:05.484138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.484125 4865 scope.go:122] "RemoveContainer" containerID="d4d6a506bf27288a7ab5ff213ec8668776905d9ea0826218e3f04f4f4ea5be9e" Oct 31 15:30:05.484363 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.484346 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr_odh-ai-gateway-infra(bef53331-8c08-41e5-81be-70ed9d941e4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" Oct 31 15:30:05.486199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.486179 4865 generic.go:356] "Generic (PLEG): container finished" podID="c45ae376-b982-439f-8d90-10b0e90974ef" containerID="e77cefae191a0ba68e0dcca852fc7eb0c716b968b788204ab3d279b66785239b" exitCode=60 Oct 31 15:30:05.486292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.486265 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" event={"ID":"c45ae376-b982-439f-8d90-10b0e90974ef","Type":"ContainerDied","Data":"e77cefae191a0ba68e0dcca852fc7eb0c716b968b788204ab3d279b66785239b"} Oct 31 15:30:05.486453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.486440 4865 scope.go:122] "RemoveContainer" containerID="e77cefae191a0ba68e0dcca852fc7eb0c716b968b788204ab3d279b66785239b" Oct 31 15:30:05.486681 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.486664 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj_odh-ai-gateway-infra(c45ae376-b982-439f-8d90-10b0e90974ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" Oct 31 15:30:05.488385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.488368 4865 generic.go:356] "Generic (PLEG): container finished" podID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerID="bcb00b699585eea03335c9a70487af131ba78d6e6bf40838b2a28d90d1c8268d" exitCode=60 Oct 31 15:30:05.488514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.488374 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" event={"ID":"76d82b74-7865-4772-be6c-38b4d7a2660b","Type":"ContainerDied","Data":"bcb00b699585eea03335c9a70487af131ba78d6e6bf40838b2a28d90d1c8268d"} Oct 31 15:30:05.488759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.488741 4865 scope.go:122] "RemoveContainer" containerID="bcb00b699585eea03335c9a70487af131ba78d6e6bf40838b2a28d90d1c8268d" Oct 31 15:30:05.489269 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.489251 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2_odh-ai-gateway-infra(76d82b74-7865-4772-be6c-38b4d7a2660b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" Oct 31 15:30:05.490662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.490644 4865 generic.go:356] "Generic (PLEG): container finished" podID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerID="1b0622c2095c8f49236db7708b4e26ce324e563f695b9fe481b1a83474e2d81a" exitCode=60 Oct 31 15:30:05.490713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.490704 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" event={"ID":"69cdc270-e7df-4e32-b486-78063e4d3de5","Type":"ContainerDied","Data":"1b0622c2095c8f49236db7708b4e26ce324e563f695b9fe481b1a83474e2d81a"} Oct 31 15:30:05.490935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.490918 4865 scope.go:122] "RemoveContainer" containerID="1b0622c2095c8f49236db7708b4e26ce324e563f695b9fe481b1a83474e2d81a" Oct 31 15:30:05.491197 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.491173 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn_odh-ai-gateway-infra(69cdc270-e7df-4e32-b486-78063e4d3de5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" Oct 31 15:30:05.491345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.491301 4865 scope.go:122] "RemoveContainer" containerID="0480dd0902c319d0f2039ab6eeef46b6d0c131662cff72462328c1440488b66f" Oct 31 15:30:05.493056 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.493042 4865 scope.go:122] "RemoveContainer" containerID="6fb0e261474fa7dbbd3d728bd47080e6917b69b0ab7407c87c5e6de92ceace74" Oct 31 15:30:05.493268 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.493252 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn_odh-ai-gateway-infra(040913dd-2664-4379-a9c3-c77b61e9c5b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" Oct 31 15:30:05.520514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.519251 4865 generic.go:356] "Generic (PLEG): container finished" podID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerID="633545669c1c9abf4e0596ace220337bcec412dbdb651ddcfd8f2a5dbbf0e645" exitCode=60 Oct 31 15:30:05.520514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.519305 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerDied","Data":"633545669c1c9abf4e0596ace220337bcec412dbdb651ddcfd8f2a5dbbf0e645"} Oct 31 15:30:05.520514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.519824 4865 scope.go:122] "RemoveContainer" containerID="633545669c1c9abf4e0596ace220337bcec412dbdb651ddcfd8f2a5dbbf0e645" Oct 31 15:30:05.520514 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:05.520249 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4_odh-ai-gateway-infra(d8fedf9b-eb43-4a28-9796-8b5545ca2ddd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" Oct 31 15:30:05.558441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.558408 4865 scope.go:122] "RemoveContainer" containerID="cef8ea724d6976f91a4249ccf11d87b4d38742a5a3dd0e7db6cc4d2b51d06cf0" Oct 31 15:30:05.573395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.573373 4865 scope.go:122] "RemoveContainer" containerID="fab7d0c945ee4a9f8f0b09c8eb34ef6f7ff1ca2baf903f5fecff02bb196eec66" Oct 31 15:30:05.589000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.588977 4865 scope.go:122] "RemoveContainer" containerID="ae06a6f67809945fa2a140f85b59353db57975f6f0457aba44dfcfb4b2160b11" Oct 31 15:30:05.633496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.632864 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:05.659976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.659291 4865 scope.go:122] "RemoveContainer" containerID="157b5cbfa8b8c3912ca37611b533303efa6380d41948ea62bac5feded8b82299" Oct 31 15:30:05.734590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.734559 4865 scope.go:122] "RemoveContainer" containerID="cfda5f561bb56347d8de3b1bb4aca95486945439efd3cccde62d214679dadcbc" Oct 31 15:30:05.777509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.777485 4865 scope.go:122] "RemoveContainer" containerID="3c02ca0764a07159a62aa33cde7d0a63dccf433fcce51dc8faace8974dcc3280" Oct 31 15:30:05.883767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:05.883729 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:06.468200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.468153 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9"] Oct 31 15:30:06.474076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.474050 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.563763 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvgl\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-kube-api-access-6xvgl\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.563828 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.563899 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.563923 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785a65e1-79df-46fd-94dc-f8118b3b2f73-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.564050 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.564085 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.564109 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.564173 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.564255 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.579827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.565256 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9"] Oct 31 15:30:06.610395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.608918 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-2ad95a-openshift-default-dockercfg-6br7x\"" Oct 31 15:30:06.622929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.622893 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:06.777071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.776976 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.777336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.777315 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.777530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.777501 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.777738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.777722 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.777906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.777892 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvgl\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-kube-api-access-6xvgl\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.778026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.778011 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.778203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.778170 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.778322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.778307 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785a65e1-79df-46fd-94dc-f8118b3b2f73-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.778577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.778563 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.779115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.779095 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.779484 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.779449 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.782211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.782175 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.789255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.789229 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.789374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.789301 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.789764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.789742 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785a65e1-79df-46fd-94dc-f8118b3b2f73-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.790127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.790105 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.854521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.850039 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.854521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.853318 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvgl\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-kube-api-access-6xvgl\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:06.865073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.865042 4865 scope.go:122] "RemoveContainer" containerID="2c3ccd80ae13885783d045e4eb762d3368ff7d310413e96a18303a997b00b3db" Oct 31 15:30:06.865505 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.865441 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t_odh-ai-gateway-infra(c16ecf29-3bde-4477-85bc-59e0291549fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" Oct 31 15:30:06.868351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.868328 4865 scope.go:122] "RemoveContainer" containerID="c868c42dbf0c3b4023e70490cceaf125362b638c1913eda67bdf7d7fad30cb7d" Oct 31 15:30:06.949574 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.942952 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg_odh-ai-gateway-infra(621c588b-3dc6-40ae-a1c2-a6ddb214669b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" Oct 31 15:30:06.949574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.943732 4865 scope.go:122] "RemoveContainer" containerID="acc420ab2c5f6fc8820785acc9c610758c2055f2b231dc3153b009a02e311abf" Oct 31 15:30:06.949574 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.944584 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw_odh-ai-gateway-infra(60b21624-f0ba-4179-827b-7dfb79f51d39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" Oct 31 15:30:06.950247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.950201 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:06.951925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.951900 4865 scope.go:122] "RemoveContainer" containerID="d4d6a506bf27288a7ab5ff213ec8668776905d9ea0826218e3f04f4f4ea5be9e" Oct 31 15:30:06.952255 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.952227 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr_odh-ai-gateway-infra(bef53331-8c08-41e5-81be-70ed9d941e4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" Oct 31 15:30:06.954758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.954742 4865 scope.go:122] "RemoveContainer" containerID="e77cefae191a0ba68e0dcca852fc7eb0c716b968b788204ab3d279b66785239b" Oct 31 15:30:06.955306 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.955272 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj_odh-ai-gateway-infra(c45ae376-b982-439f-8d90-10b0e90974ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" Oct 31 15:30:06.955801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.955783 4865 scope.go:122] "RemoveContainer" containerID="bcb00b699585eea03335c9a70487af131ba78d6e6bf40838b2a28d90d1c8268d" Oct 31 15:30:06.956221 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.956191 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2_odh-ai-gateway-infra(76d82b74-7865-4772-be6c-38b4d7a2660b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" Oct 31 15:30:06.958726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.958703 4865 scope.go:122] "RemoveContainer" containerID="1b0622c2095c8f49236db7708b4e26ce324e563f695b9fe481b1a83474e2d81a" Oct 31 15:30:06.959008 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.958990 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn_odh-ai-gateway-infra(69cdc270-e7df-4e32-b486-78063e4d3de5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" Oct 31 15:30:06.959072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:06.959031 4865 scope.go:122] "RemoveContainer" containerID="633545669c1c9abf4e0596ace220337bcec412dbdb651ddcfd8f2a5dbbf0e645" Oct 31 15:30:06.959250 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:06.959233 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4_odh-ai-gateway-infra(d8fedf9b-eb43-4a28-9796-8b5545ca2ddd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" Oct 31 15:30:07.089415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:07.089343 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:07.633843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:07.629941 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9"] Oct 31 15:30:07.651062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:07.651021 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:07.651196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:07.651110 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:07.651196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:07.651152 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:07.970729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:07.970684 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" event={"ID":"785a65e1-79df-46fd-94dc-f8118b3b2f73","Type":"ContainerStarted","Data":"4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a"} Oct 31 15:30:07.970891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:07.970747 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" event={"ID":"785a65e1-79df-46fd-94dc-f8118b3b2f73","Type":"ContainerStarted","Data":"9b89789223b13447d0373155c6a3d9e88f58718276de59191b79eb85f1934640"} Oct 31 15:30:08.089915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:08.089880 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:08.091332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:08.091301 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 15:30:08.091449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:08.091362 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 15:30:08.285594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:08.285451 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podStartSLOduration=2.28543252 podStartE2EDuration="2.28543252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:08.273771626 +0000 UTC m=+2653.893323047" watchObservedRunningTime="2026-10-31 15:30:08.28543252 +0000 UTC m=+2653.904983933" Oct 31 15:30:09.095519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:09.095482 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:09.991679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:09.991635 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:09.992747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:09.992713 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:30:10.412002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:10.411926 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:10.995961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:10.995921 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:11.175640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:11.175608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:11.315445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:11.315355 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:12.009660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:12.009630 4865 generic.go:356] "Generic (PLEG): container finished" podID="cf89f0e4-3499-4802-9a05-72062f42def8" containerID="2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754" exitCode=0 Oct 31 15:30:12.010128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:12.009706 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" event={"ID":"cf89f0e4-3499-4802-9a05-72062f42def8","Type":"ContainerDied","Data":"2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754"} Oct 31 15:30:12.010512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:12.010459 4865 scope.go:122] "RemoveContainer" containerID="2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754" Oct 31 15:30:12.012147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:12.012113 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:12.012242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:12.012210 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:12.012297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:12.012282 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:13.027019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.026986 4865 generic.go:356] "Generic (PLEG): container finished" podID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerID="bd51cd0fecb5c65f1f7c455ff55ebffc7c12d2e280c649e8be8ed8704c7bac1c" exitCode=0 Oct 31 15:30:13.027570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.027068 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" event={"ID":"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1","Type":"ContainerDied","Data":"bd51cd0fecb5c65f1f7c455ff55ebffc7c12d2e280c649e8be8ed8704c7bac1c"} Oct 31 15:30:13.027655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.027620 4865 scope.go:122] "RemoveContainer" containerID="bd51cd0fecb5c65f1f7c455ff55ebffc7c12d2e280c649e8be8ed8704c7bac1c" Oct 31 15:30:13.029335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.029312 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" event={"ID":"cf89f0e4-3499-4802-9a05-72062f42def8","Type":"ContainerStarted","Data":"fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8"} Oct 31 15:30:13.029445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.029305 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:13.029536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.029460 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:13.029536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.029530 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:13.071782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.071763 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:30:13.597904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.597872 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:30:13.612680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.612623 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 31 15:30:13.613085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:13.613031 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 31 15:30:14.056744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.056707 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" event={"ID":"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1","Type":"ContainerStarted","Data":"d1b6aade064ad7c4a53c1d563c4c156415727ada83484d664a2cf14df243a1d9"} Oct 31 15:30:14.180027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.179998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:14.404645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.381000 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:14.637496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.637199 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 31 15:30:14.637496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.637270 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 31 15:30:14.678276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.678170 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:30:14.679706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.679678 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 31 15:30:14.679869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.679740 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 31 15:30:14.803494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.803031 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n"] Oct 31 15:30:14.842498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.839121 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:14.866926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.855668 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n"] Oct 31 15:30:14.894824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.894787 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-2ad95a\"" Oct 31 15:30:14.895253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.895233 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-2ad95a-dockercfg-h7jx4\"" Oct 31 15:30:14.960956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.960924 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8"] Oct 31 15:30:14.966897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:14.966873 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.049692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.049654 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnh8\" (UniqueName: \"kubernetes.io/projected/80207be4-6a62-423c-9691-a64155ace7a9-kube-api-access-klnh8\") pod \"payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:15.050376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.050343 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80207be4-6a62-423c-9691-a64155ace7a9-plugins-config-volume\") pod \"payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:15.053350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.053324 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8"] Oct 31 15:30:15.152125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.151695 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.152125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.151830 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80207be4-6a62-423c-9691-a64155ace7a9-plugins-config-volume\") pod \"payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:15.152125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.151876 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghhf\" (UniqueName: \"kubernetes.io/projected/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-kube-api-access-cghhf\") pod \"payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.152125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.152078 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnh8\" (UniqueName: \"kubernetes.io/projected/80207be4-6a62-423c-9691-a64155ace7a9-kube-api-access-klnh8\") pod \"payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:15.153073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.153053 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80207be4-6a62-423c-9691-a64155ace7a9-plugins-config-volume\") pod \"payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:15.249538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.249303 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnh8\" (UniqueName: \"kubernetes.io/projected/80207be4-6a62-423c-9691-a64155ace7a9-kube-api-access-klnh8\") pod \"payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:15.253711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.253683 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.253880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.253859 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghhf\" (UniqueName: \"kubernetes.io/projected/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-kube-api-access-cghhf\") pod \"payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.254925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.254901 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.337210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.337173 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghhf\" (UniqueName: \"kubernetes.io/projected/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-kube-api-access-cghhf\") pod \"payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.472127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.472090 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:15.489990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.489959 4865 scope.go:122] "RemoveContainer" containerID="6fb0e261474fa7dbbd3d728bd47080e6917b69b0ab7407c87c5e6de92ceace74" Oct 31 15:30:15.491266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.491247 4865 scope.go:122] "RemoveContainer" containerID="2c3ccd80ae13885783d045e4eb762d3368ff7d310413e96a18303a997b00b3db" Oct 31 15:30:15.581636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.581594 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:15.668443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.640810 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 31 15:30:15.668443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.640867 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 31 15:30:15.687146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.686636 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 31 15:30:15.687146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.686683 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 31 15:30:15.688844 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:15.688814 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:16.055295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.054999 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97"] Oct 31 15:30:16.086769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.086708 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b238505e-j2m97"] Oct 31 15:30:16.107074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.093792 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" event={"ID":"040913dd-2664-4379-a9c3-c77b61e9c5b0","Type":"ContainerStarted","Data":"bd759f7e28f4a8d38b0a5566db0951e3b5dfb936f69e28f1717206f0358668b0"} Oct 31 15:30:16.168981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.168771 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n"] Oct 31 15:30:16.169520 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:30:16.169459 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80207be4_6a62_423c_9691_a64155ace7a9.slice/crio-c1e3dc5a529639c953f2bc2243b015ef46a50960279de58dc6ce057f9811dcf8 WatchSource:0}: Error finding container c1e3dc5a529639c953f2bc2243b015ef46a50960279de58dc6ce057f9811dcf8: Status 404 returned error can't find the container with id c1e3dc5a529639c953f2bc2243b015ef46a50960279de58dc6ce057f9811dcf8 Oct 31 15:30:16.258535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.258501 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8"] Oct 31 15:30:16.489998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.489968 4865 scope.go:122] "RemoveContainer" containerID="e77cefae191a0ba68e0dcca852fc7eb0c716b968b788204ab3d279b66785239b" Oct 31 15:30:16.490582 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.490558 4865 scope.go:122] "RemoveContainer" containerID="c868c42dbf0c3b4023e70490cceaf125362b638c1913eda67bdf7d7fad30cb7d" Oct 31 15:30:16.491137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.490773 4865 scope.go:122] "RemoveContainer" containerID="acc420ab2c5f6fc8820785acc9c610758c2055f2b231dc3153b009a02e311abf" Oct 31 15:30:16.550494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.549706 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd" path="/var/lib/kubelet/pods/f78f3bd9-328d-42eb-8a3a-e87ba6e2f9fd/volumes" Oct 31 15:30:16.618928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.601576 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 31 15:30:16.618928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.601635 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 31 15:30:16.678185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.678159 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 31 15:30:16.678356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:16.678202 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 31 15:30:17.156548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.156511 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" event={"ID":"80207be4-6a62-423c-9691-a64155ace7a9","Type":"ContainerStarted","Data":"ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97"} Oct 31 15:30:17.156643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.156580 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" event={"ID":"80207be4-6a62-423c-9691-a64155ace7a9","Type":"ContainerStarted","Data":"c1e3dc5a529639c953f2bc2243b015ef46a50960279de58dc6ce057f9811dcf8"} Oct 31 15:30:17.158150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.157886 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:17.158801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.158776 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" event={"ID":"b528266d-5fc3-4ccb-9e7a-a53fe13f15de","Type":"ContainerStarted","Data":"625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a"} Oct 31 15:30:17.159004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.158969 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" event={"ID":"b528266d-5fc3-4ccb-9e7a-a53fe13f15de","Type":"ContainerStarted","Data":"07d0f4757897243248df99212174c54ad8f1b2f1ef3f8b254f532490a55c309d"} Oct 31 15:30:17.159146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.159130 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:17.161235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.161211 4865 generic.go:356] "Generic (PLEG): container finished" podID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerID="0eaf904dd4651e399ad9440dc2505e95a76a16ded4526b1ddf7a290e64dd0339" exitCode=60 Oct 31 15:30:17.161337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.161270 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" event={"ID":"c16ecf29-3bde-4477-85bc-59e0291549fa","Type":"ContainerDied","Data":"0eaf904dd4651e399ad9440dc2505e95a76a16ded4526b1ddf7a290e64dd0339"} Oct 31 15:30:17.161337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.161324 4865 scope.go:122] "RemoveContainer" containerID="2c3ccd80ae13885783d045e4eb762d3368ff7d310413e96a18303a997b00b3db" Oct 31 15:30:17.161664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.161649 4865 scope.go:122] "RemoveContainer" containerID="0eaf904dd4651e399ad9440dc2505e95a76a16ded4526b1ddf7a290e64dd0339" Oct 31 15:30:17.161936 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:17.161917 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t_odh-ai-gateway-infra(c16ecf29-3bde-4477-85bc-59e0291549fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" Oct 31 15:30:17.164264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.164242 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerStarted","Data":"8c2082242d9a7ee4b77117170818b74dbb8ec95739a5fa76466dda380bb4f6ed"} Oct 31 15:30:17.166013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.165993 4865 generic.go:356] "Generic (PLEG): container finished" podID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerID="bd759f7e28f4a8d38b0a5566db0951e3b5dfb936f69e28f1717206f0358668b0" exitCode=60 Oct 31 15:30:17.166118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.166022 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" event={"ID":"040913dd-2664-4379-a9c3-c77b61e9c5b0","Type":"ContainerDied","Data":"bd759f7e28f4a8d38b0a5566db0951e3b5dfb936f69e28f1717206f0358668b0"} Oct 31 15:30:17.166334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.166316 4865 scope.go:122] "RemoveContainer" containerID="bd759f7e28f4a8d38b0a5566db0951e3b5dfb936f69e28f1717206f0358668b0" Oct 31 15:30:17.166593 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:17.166572 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn_odh-ai-gateway-infra(040913dd-2664-4379-a9c3-c77b61e9c5b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" Oct 31 15:30:17.175907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.175887 4865 scope.go:122] "RemoveContainer" containerID="6fb0e261474fa7dbbd3d728bd47080e6917b69b0ab7407c87c5e6de92ceace74" Oct 31 15:30:17.253699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.253672 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn"] Oct 31 15:30:17.283908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.283850 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" podStartSLOduration=3.283836509 podStartE2EDuration="3.283836509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:17.279685294 +0000 UTC m=+2662.899236709" watchObservedRunningTime="2026-10-31 15:30:17.283836509 +0000 UTC m=+2662.903387919" Oct 31 15:30:17.489225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.489185 4865 scope.go:122] "RemoveContainer" containerID="1b0622c2095c8f49236db7708b4e26ce324e563f695b9fe481b1a83474e2d81a" Oct 31 15:30:17.489449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.489421 4865 scope.go:122] "RemoveContainer" containerID="bcb00b699585eea03335c9a70487af131ba78d6e6bf40838b2a28d90d1c8268d" Oct 31 15:30:17.489774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.489730 4865 scope.go:122] "RemoveContainer" containerID="d4d6a506bf27288a7ab5ff213ec8668776905d9ea0826218e3f04f4f4ea5be9e" Oct 31 15:30:17.545497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.542827 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" podStartSLOduration=3.54280568 podStartE2EDuration="3.54280568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:17.53227424 +0000 UTC m=+2663.151825676" watchObservedRunningTime="2026-10-31 15:30:17.54280568 +0000 UTC m=+2663.162357092" Oct 31 15:30:17.557790 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:17.557751 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:30:17.619945 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.619912 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 31 15:30:17.620256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.620224 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 31 15:30:17.691491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.687964 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 31 15:30:17.691491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:17.688021 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 31 15:30:18.176734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.176704 4865 generic.go:356] "Generic (PLEG): container finished" podID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerID="144492e23a6b7dfdf54b41867c71e2c2d4f3da1c988a3536ccff0dfccca2a587" exitCode=60 Oct 31 15:30:18.177191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.176777 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" event={"ID":"60b21624-f0ba-4179-827b-7dfb79f51d39","Type":"ContainerDied","Data":"144492e23a6b7dfdf54b41867c71e2c2d4f3da1c988a3536ccff0dfccca2a587"} Oct 31 15:30:18.177286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.177228 4865 scope.go:122] "RemoveContainer" containerID="acc420ab2c5f6fc8820785acc9c610758c2055f2b231dc3153b009a02e311abf" Oct 31 15:30:18.177733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.177712 4865 scope.go:122] "RemoveContainer" containerID="144492e23a6b7dfdf54b41867c71e2c2d4f3da1c988a3536ccff0dfccca2a587" Oct 31 15:30:18.178173 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:18.178140 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw_odh-ai-gateway-infra(60b21624-f0ba-4179-827b-7dfb79f51d39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" Oct 31 15:30:18.179740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.179717 4865 generic.go:356] "Generic (PLEG): container finished" podID="c45ae376-b982-439f-8d90-10b0e90974ef" containerID="0564750f68f087bb6157bfad267ca4c68c1c8fc0809b26c46436578c1cb50598" exitCode=60 Oct 31 15:30:18.179835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.179824 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" event={"ID":"c45ae376-b982-439f-8d90-10b0e90974ef","Type":"ContainerDied","Data":"0564750f68f087bb6157bfad267ca4c68c1c8fc0809b26c46436578c1cb50598"} Oct 31 15:30:18.180128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.180112 4865 scope.go:122] "RemoveContainer" containerID="0564750f68f087bb6157bfad267ca4c68c1c8fc0809b26c46436578c1cb50598" Oct 31 15:30:18.180353 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:18.180338 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj_odh-ai-gateway-infra(c45ae376-b982-439f-8d90-10b0e90974ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" Oct 31 15:30:18.185516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.185495 4865 generic.go:356] "Generic (PLEG): container finished" podID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerID="8c2082242d9a7ee4b77117170818b74dbb8ec95739a5fa76466dda380bb4f6ed" exitCode=60 Oct 31 15:30:18.186000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.185978 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerDied","Data":"8c2082242d9a7ee4b77117170818b74dbb8ec95739a5fa76466dda380bb4f6ed"} Oct 31 15:30:18.186238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.186213 4865 scope.go:122] "RemoveContainer" containerID="8c2082242d9a7ee4b77117170818b74dbb8ec95739a5fa76466dda380bb4f6ed" Oct 31 15:30:18.186421 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:18.186404 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg_odh-ai-gateway-infra(621c588b-3dc6-40ae-a1c2-a6ddb214669b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" Oct 31 15:30:18.192564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.192542 4865 scope.go:122] "RemoveContainer" containerID="e77cefae191a0ba68e0dcca852fc7eb0c716b968b788204ab3d279b66785239b" Oct 31 15:30:18.250160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.249123 4865 scope.go:122] "RemoveContainer" containerID="c868c42dbf0c3b4023e70490cceaf125362b638c1913eda67bdf7d7fad30cb7d" Oct 31 15:30:18.381660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.381617 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:30:18.439873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.439849 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg"] Oct 31 15:30:18.472142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.472107 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:18.489723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.489702 4865 scope.go:122] "RemoveContainer" containerID="633545669c1c9abf4e0596ace220337bcec412dbdb651ddcfd8f2a5dbbf0e645" Oct 31 15:30:18.586020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.585988 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040913dd-2664-4379-a9c3-c77b61e9c5b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/040913dd-2664-4379-a9c3-c77b61e9c5b0-maas-api-service-ca\") pod \"040913dd-2664-4379-a9c3-c77b61e9c5b0\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " Oct 31 15:30:18.586175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.586054 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040913dd-2664-4379-a9c3-c77b61e9c5b0-kube-api-access-d8zv5\" (UniqueName: \"kubernetes.io/projected/040913dd-2664-4379-a9c3-c77b61e9c5b0-kube-api-access-d8zv5\") pod \"040913dd-2664-4379-a9c3-c77b61e9c5b0\" (UID: \"040913dd-2664-4379-a9c3-c77b61e9c5b0\") " Oct 31 15:30:18.586373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.586347 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040913dd-2664-4379-a9c3-c77b61e9c5b0-maas-api-service-ca" pod "040913dd-2664-4379-a9c3-c77b61e9c5b0" (UID: "040913dd-2664-4379-a9c3-c77b61e9c5b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:18.587119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.586848 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/040913dd-2664-4379-a9c3-c77b61e9c5b0-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:18.588424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.588399 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040913dd-2664-4379-a9c3-c77b61e9c5b0-kube-api-access-d8zv5" pod "040913dd-2664-4379-a9c3-c77b61e9c5b0" (UID: "040913dd-2664-4379-a9c3-c77b61e9c5b0"). InnerVolumeSpecName "kube-api-access-d8zv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:18.631134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.631091 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:30:18.634013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.633982 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 15:30:18.678600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.678570 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Oct 31 15:30:18.678771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.678623 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Oct 31 15:30:18.688755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.688686 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8zv5\" (UniqueName: \"kubernetes.io/projected/040913dd-2664-4379-a9c3-c77b61e9c5b0-kube-api-access-d8zv5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:18.783240 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:18.783208 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t"] Oct 31 15:30:19.127349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.127309 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:19.218610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.218283 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" event={"ID":"c16ecf29-3bde-4477-85bc-59e0291549fa","Type":"ContainerDied","Data":"1b5ffc479470becd6dbd81ec1178bf288b9caad03c3aea78cae67d14eb9c7fc4"} Oct 31 15:30:19.218610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.218361 4865 scope.go:122] "RemoveContainer" containerID="0eaf904dd4651e399ad9440dc2505e95a76a16ded4526b1ddf7a290e64dd0339" Oct 31 15:30:19.218610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.218525 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t" Oct 31 15:30:19.226826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.226770 4865 generic.go:356] "Generic (PLEG): container finished" podID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerID="78d35b100c34accfd86723f72434467847a25794192c6cd429510bfdbd47c0ec" exitCode=60 Oct 31 15:30:19.226936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.226838 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" event={"ID":"bef53331-8c08-41e5-81be-70ed9d941e4b","Type":"ContainerDied","Data":"78d35b100c34accfd86723f72434467847a25794192c6cd429510bfdbd47c0ec"} Oct 31 15:30:19.227267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.227249 4865 scope.go:122] "RemoveContainer" containerID="78d35b100c34accfd86723f72434467847a25794192c6cd429510bfdbd47c0ec" Oct 31 15:30:19.227660 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:19.227634 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr_odh-ai-gateway-infra(bef53331-8c08-41e5-81be-70ed9d941e4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" Oct 31 15:30:19.231048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.231018 4865 generic.go:356] "Generic (PLEG): container finished" podID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerID="0c31e7249d22eb56e97e7c715b01f1b14c0e7646647530c62ae73ea2f7eade1e" exitCode=60 Oct 31 15:30:19.231147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.231085 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" event={"ID":"76d82b74-7865-4772-be6c-38b4d7a2660b","Type":"ContainerDied","Data":"0c31e7249d22eb56e97e7c715b01f1b14c0e7646647530c62ae73ea2f7eade1e"} Oct 31 15:30:19.231529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.231507 4865 scope.go:122] "RemoveContainer" containerID="0c31e7249d22eb56e97e7c715b01f1b14c0e7646647530c62ae73ea2f7eade1e" Oct 31 15:30:19.231905 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:19.231883 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2_odh-ai-gateway-infra(76d82b74-7865-4772-be6c-38b4d7a2660b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" Oct 31 15:30:19.232436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.232410 4865 scope.go:122] "RemoveContainer" containerID="d4d6a506bf27288a7ab5ff213ec8668776905d9ea0826218e3f04f4f4ea5be9e" Oct 31 15:30:19.233580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.233453 4865 generic.go:356] "Generic (PLEG): container finished" podID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerID="13460b793d1366032be6ee7467bfd2efb14d31ae5891967141673e775dbdae21" exitCode=60 Oct 31 15:30:19.233580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.233494 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" event={"ID":"69cdc270-e7df-4e32-b486-78063e4d3de5","Type":"ContainerDied","Data":"13460b793d1366032be6ee7467bfd2efb14d31ae5891967141673e775dbdae21"} Oct 31 15:30:19.233927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.233908 4865 scope.go:122] "RemoveContainer" containerID="13460b793d1366032be6ee7467bfd2efb14d31ae5891967141673e775dbdae21" Oct 31 15:30:19.234289 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:19.234265 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn_odh-ai-gateway-infra(69cdc270-e7df-4e32-b486-78063e4d3de5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" Oct 31 15:30:19.234512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.234489 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16ecf29-3bde-4477-85bc-59e0291549fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16ecf29-3bde-4477-85bc-59e0291549fa-maas-api-service-ca\") pod \"c16ecf29-3bde-4477-85bc-59e0291549fa\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " Oct 31 15:30:19.234605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.234547 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16ecf29-3bde-4477-85bc-59e0291549fa-kube-api-access-lgp72\" (UniqueName: \"kubernetes.io/projected/c16ecf29-3bde-4477-85bc-59e0291549fa-kube-api-access-lgp72\") pod \"c16ecf29-3bde-4477-85bc-59e0291549fa\" (UID: \"c16ecf29-3bde-4477-85bc-59e0291549fa\") " Oct 31 15:30:19.234844 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.234819 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16ecf29-3bde-4477-85bc-59e0291549fa-maas-api-service-ca" pod "c16ecf29-3bde-4477-85bc-59e0291549fa" (UID: "c16ecf29-3bde-4477-85bc-59e0291549fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:19.235488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.235166 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" Oct 31 15:30:19.235488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.235197 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16ecf29-3bde-4477-85bc-59e0291549fa-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.235488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.235213 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn" event={"ID":"040913dd-2664-4379-a9c3-c77b61e9c5b0","Type":"ContainerDied","Data":"a39685a333946c39295d8f6ea53064ca583698283b3d50f76654d81fa1e5d32c"} Oct 31 15:30:19.236865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.236845 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16ecf29-3bde-4477-85bc-59e0291549fa-kube-api-access-lgp72" pod "c16ecf29-3bde-4477-85bc-59e0291549fa" (UID: "c16ecf29-3bde-4477-85bc-59e0291549fa"). InnerVolumeSpecName "kube-api-access-lgp72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:19.238146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.238122 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerStarted","Data":"424a0eb9b2f22b872a1dbf08b55c48e8380a6f185f3f7afcab7964b8d798eaf6"} Oct 31 15:30:19.247831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.247808 4865 scope.go:122] "RemoveContainer" containerID="bcb00b699585eea03335c9a70487af131ba78d6e6bf40838b2a28d90d1c8268d" Oct 31 15:30:19.262038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.262017 4865 scope.go:122] "RemoveContainer" containerID="1b0622c2095c8f49236db7708b4e26ce324e563f695b9fe481b1a83474e2d81a" Oct 31 15:30:19.285100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.285076 4865 scope.go:122] "RemoveContainer" containerID="bd759f7e28f4a8d38b0a5566db0951e3b5dfb936f69e28f1717206f0358668b0" Oct 31 15:30:19.307962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.307889 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw"] Oct 31 15:30:19.342460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.342402 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgp72\" (UniqueName: \"kubernetes.io/projected/c16ecf29-3bde-4477-85bc-59e0291549fa-kube-api-access-lgp72\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.458116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.458084 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:19.467995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.467969 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj"] Oct 31 15:30:19.543817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.540899 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:19.549071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.549045 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621c588b-3dc6-40ae-a1c2-a6ddb214669b-kube-api-access-rhrr6\" (UniqueName: \"kubernetes.io/projected/621c588b-3dc6-40ae-a1c2-a6ddb214669b-kube-api-access-rhrr6\") pod \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " Oct 31 15:30:19.549213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.549145 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621c588b-3dc6-40ae-a1c2-a6ddb214669b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621c588b-3dc6-40ae-a1c2-a6ddb214669b-maas-api-service-ca\") pod \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\" (UID: \"621c588b-3dc6-40ae-a1c2-a6ddb214669b\") " Oct 31 15:30:19.551699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.550172 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621c588b-3dc6-40ae-a1c2-a6ddb214669b-maas-api-service-ca" pod "621c588b-3dc6-40ae-a1c2-a6ddb214669b" (UID: "621c588b-3dc6-40ae-a1c2-a6ddb214669b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:19.594382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.594119 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621c588b-3dc6-40ae-a1c2-a6ddb214669b-kube-api-access-rhrr6" pod "621c588b-3dc6-40ae-a1c2-a6ddb214669b" (UID: "621c588b-3dc6-40ae-a1c2-a6ddb214669b"). InnerVolumeSpecName "kube-api-access-rhrr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:19.650659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.650624 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b21624-f0ba-4179-827b-7dfb79f51d39-kube-api-access-mrsqh\" (UniqueName: \"kubernetes.io/projected/60b21624-f0ba-4179-827b-7dfb79f51d39-kube-api-access-mrsqh\") pod \"60b21624-f0ba-4179-827b-7dfb79f51d39\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " Oct 31 15:30:19.650855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.650684 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b21624-f0ba-4179-827b-7dfb79f51d39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b21624-f0ba-4179-827b-7dfb79f51d39-maas-api-service-ca\") pod \"60b21624-f0ba-4179-827b-7dfb79f51d39\" (UID: \"60b21624-f0ba-4179-827b-7dfb79f51d39\") " Oct 31 15:30:19.651375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.651351 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/621c588b-3dc6-40ae-a1c2-a6ddb214669b-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.651510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.651382 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrr6\" (UniqueName: \"kubernetes.io/projected/621c588b-3dc6-40ae-a1c2-a6ddb214669b-kube-api-access-rhrr6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.651788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.651766 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b21624-f0ba-4179-827b-7dfb79f51d39-maas-api-service-ca" pod "60b21624-f0ba-4179-827b-7dfb79f51d39" (UID: "60b21624-f0ba-4179-827b-7dfb79f51d39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:19.654792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.654761 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b21624-f0ba-4179-827b-7dfb79f51d39-kube-api-access-mrsqh" pod "60b21624-f0ba-4179-827b-7dfb79f51d39" (UID: "60b21624-f0ba-4179-827b-7dfb79f51d39"). InnerVolumeSpecName "kube-api-access-mrsqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:19.687710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.687683 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:30:19.688907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.688886 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 15:30:19.696087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.696067 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:19.752715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.752649 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrsqh\" (UniqueName: \"kubernetes.io/projected/60b21624-f0ba-4179-827b-7dfb79f51d39-kube-api-access-mrsqh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.752715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.752672 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b21624-f0ba-4179-827b-7dfb79f51d39-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.853379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.853352 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45ae376-b982-439f-8d90-10b0e90974ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45ae376-b982-439f-8d90-10b0e90974ef-maas-api-service-ca\") pod \"c45ae376-b982-439f-8d90-10b0e90974ef\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " Oct 31 15:30:19.853586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.853506 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45ae376-b982-439f-8d90-10b0e90974ef-kube-api-access-7xn4n\" (UniqueName: \"kubernetes.io/projected/c45ae376-b982-439f-8d90-10b0e90974ef-kube-api-access-7xn4n\") pod \"c45ae376-b982-439f-8d90-10b0e90974ef\" (UID: \"c45ae376-b982-439f-8d90-10b0e90974ef\") " Oct 31 15:30:19.853707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.853685 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45ae376-b982-439f-8d90-10b0e90974ef-maas-api-service-ca" pod "c45ae376-b982-439f-8d90-10b0e90974ef" (UID: "c45ae376-b982-439f-8d90-10b0e90974ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:19.854128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.854097 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45ae376-b982-439f-8d90-10b0e90974ef-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.855782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.855762 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45ae376-b982-439f-8d90-10b0e90974ef-kube-api-access-7xn4n" pod "c45ae376-b982-439f-8d90-10b0e90974ef" (UID: "c45ae376-b982-439f-8d90-10b0e90974ef"). InnerVolumeSpecName "kube-api-access-7xn4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:19.955327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.955300 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xn4n\" (UniqueName: \"kubernetes.io/projected/c45ae376-b982-439f-8d90-10b0e90974ef-kube-api-access-7xn4n\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:19.981093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.981070 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn"] Oct 31 15:30:19.985378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:19.985356 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891010-qgjmn"] Oct 31 15:30:20.250079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.250045 4865 generic.go:356] "Generic (PLEG): container finished" podID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerID="424a0eb9b2f22b872a1dbf08b55c48e8380a6f185f3f7afcab7964b8d798eaf6" exitCode=60 Oct 31 15:30:20.250590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.250120 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerDied","Data":"424a0eb9b2f22b872a1dbf08b55c48e8380a6f185f3f7afcab7964b8d798eaf6"} Oct 31 15:30:20.250590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.250201 4865 scope.go:122] "RemoveContainer" containerID="633545669c1c9abf4e0596ace220337bcec412dbdb651ddcfd8f2a5dbbf0e645" Oct 31 15:30:20.250704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.250595 4865 scope.go:122] "RemoveContainer" containerID="424a0eb9b2f22b872a1dbf08b55c48e8380a6f185f3f7afcab7964b8d798eaf6" Oct 31 15:30:20.251031 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:20.251009 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4_odh-ai-gateway-infra(d8fedf9b-eb43-4a28-9796-8b5545ca2ddd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" Oct 31 15:30:20.252685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.252669 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" Oct 31 15:30:20.252792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.252669 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg" event={"ID":"621c588b-3dc6-40ae-a1c2-a6ddb214669b","Type":"ContainerDied","Data":"80cc6d2a9ed6d3eaa2bf6b26cf65f1349eb843c4ac0693a91a3920983997b0f5"} Oct 31 15:30:20.254079 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.254054 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" event={"ID":"60b21624-f0ba-4179-827b-7dfb79f51d39","Type":"ContainerDied","Data":"e88f0c8b829345012cb904d3e20b9a20feef534021cb9863a282d58cc980665e"} Oct 31 15:30:20.254188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.254094 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw" Oct 31 15:30:20.257513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.257492 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" Oct 31 15:30:20.257513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.257503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj" event={"ID":"c45ae376-b982-439f-8d90-10b0e90974ef","Type":"ContainerDied","Data":"1de9925405cb664fe2be22d58865aa5bf0b32c8bb4f14dc86287713b7502715b"} Oct 31 15:30:20.279228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.279202 4865 scope.go:122] "RemoveContainer" containerID="8c2082242d9a7ee4b77117170818b74dbb8ec95739a5fa76466dda380bb4f6ed" Oct 31 15:30:20.299388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.299364 4865 scope.go:122] "RemoveContainer" containerID="144492e23a6b7dfdf54b41867c71e2c2d4f3da1c988a3536ccff0dfccca2a587" Oct 31 15:30:20.317885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.317864 4865 scope.go:122] "RemoveContainer" containerID="0564750f68f087bb6157bfad267ca4c68c1c8fc0809b26c46436578c1cb50598" Oct 31 15:30:20.393888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.393862 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t"] Oct 31 15:30:20.399893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.399869 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891010-lw59t"] Oct 31 15:30:20.411967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.411946 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn"] Oct 31 15:30:20.502252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.502175 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" path="/var/lib/kubelet/pods/040913dd-2664-4379-a9c3-c77b61e9c5b0/volumes" Oct 31 15:30:20.502714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.502697 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" path="/var/lib/kubelet/pods/c16ecf29-3bde-4477-85bc-59e0291549fa/volumes" Oct 31 15:30:20.592204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.592173 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:20.647858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.647800 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2"] Oct 31 15:30:20.710030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.710003 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:20.752231 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.752178 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw"] Oct 31 15:30:20.799849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.784503 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69cdc270-e7df-4e32-b486-78063e4d3de5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cdc270-e7df-4e32-b486-78063e4d3de5-maas-api-service-ca\") pod \"69cdc270-e7df-4e32-b486-78063e4d3de5\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " Oct 31 15:30:20.799849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.784609 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cdc270-e7df-4e32-b486-78063e4d3de5-kube-api-access-rfm7c\" (UniqueName: \"kubernetes.io/projected/69cdc270-e7df-4e32-b486-78063e4d3de5-kube-api-access-rfm7c\") pod \"69cdc270-e7df-4e32-b486-78063e4d3de5\" (UID: \"69cdc270-e7df-4e32-b486-78063e4d3de5\") " Oct 31 15:30:20.799849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.788790 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891010-lwfbw"] Oct 31 15:30:20.799849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.789409 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cdc270-e7df-4e32-b486-78063e4d3de5-maas-api-service-ca" pod "69cdc270-e7df-4e32-b486-78063e4d3de5" (UID: "69cdc270-e7df-4e32-b486-78063e4d3de5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:20.799849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.795767 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr"] Oct 31 15:30:20.816537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.816502 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cdc270-e7df-4e32-b486-78063e4d3de5-kube-api-access-rfm7c" pod "69cdc270-e7df-4e32-b486-78063e4d3de5" (UID: "69cdc270-e7df-4e32-b486-78063e4d3de5"). InnerVolumeSpecName "kube-api-access-rfm7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:20.866761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.866736 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg"] Oct 31 15:30:20.867354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.867338 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:20.870884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.870858 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891010-54mlg"] Oct 31 15:30:20.884361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.883846 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4"] Oct 31 15:30:20.886680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.886628 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cdc270-e7df-4e32-b486-78063e4d3de5-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:20.886680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.886657 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfm7c\" (UniqueName: \"kubernetes.io/projected/69cdc270-e7df-4e32-b486-78063e4d3de5-kube-api-access-rfm7c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:20.988169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.988139 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d82b74-7865-4772-be6c-38b4d7a2660b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d82b74-7865-4772-be6c-38b4d7a2660b-maas-api-service-ca\") pod \"76d82b74-7865-4772-be6c-38b4d7a2660b\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " Oct 31 15:30:20.988332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.988251 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d82b74-7865-4772-be6c-38b4d7a2660b-kube-api-access-fqz4m\" (UniqueName: \"kubernetes.io/projected/76d82b74-7865-4772-be6c-38b4d7a2660b-kube-api-access-fqz4m\") pod \"76d82b74-7865-4772-be6c-38b4d7a2660b\" (UID: \"76d82b74-7865-4772-be6c-38b4d7a2660b\") " Oct 31 15:30:20.988495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.988457 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d82b74-7865-4772-be6c-38b4d7a2660b-maas-api-service-ca" pod "76d82b74-7865-4772-be6c-38b4d7a2660b" (UID: "76d82b74-7865-4772-be6c-38b4d7a2660b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:20.989187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.989162 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d82b74-7865-4772-be6c-38b4d7a2660b-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:20.990432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.990411 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:20.990550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:20.990505 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d82b74-7865-4772-be6c-38b4d7a2660b-kube-api-access-fqz4m" pod "76d82b74-7865-4772-be6c-38b4d7a2660b" (UID: "76d82b74-7865-4772-be6c-38b4d7a2660b"). InnerVolumeSpecName "kube-api-access-fqz4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:21.089845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.089760 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef53331-8c08-41e5-81be-70ed9d941e4b-kube-api-access-wgtp6\" (UniqueName: \"kubernetes.io/projected/bef53331-8c08-41e5-81be-70ed9d941e4b-kube-api-access-wgtp6\") pod \"bef53331-8c08-41e5-81be-70ed9d941e4b\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " Oct 31 15:30:21.090021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.089890 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef53331-8c08-41e5-81be-70ed9d941e4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef53331-8c08-41e5-81be-70ed9d941e4b-maas-api-service-ca\") pod \"bef53331-8c08-41e5-81be-70ed9d941e4b\" (UID: \"bef53331-8c08-41e5-81be-70ed9d941e4b\") " Oct 31 15:30:21.090242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.090209 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef53331-8c08-41e5-81be-70ed9d941e4b-maas-api-service-ca" pod "bef53331-8c08-41e5-81be-70ed9d941e4b" (UID: "bef53331-8c08-41e5-81be-70ed9d941e4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:21.090566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.090550 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqz4m\" (UniqueName: \"kubernetes.io/projected/76d82b74-7865-4772-be6c-38b4d7a2660b-kube-api-access-fqz4m\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:21.090611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.090568 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef53331-8c08-41e5-81be-70ed9d941e4b-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:21.092278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.092242 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef53331-8c08-41e5-81be-70ed9d941e4b-kube-api-access-wgtp6" pod "bef53331-8c08-41e5-81be-70ed9d941e4b" (UID: "bef53331-8c08-41e5-81be-70ed9d941e4b"). InnerVolumeSpecName "kube-api-access-wgtp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:21.093497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.093449 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj"] Oct 31 15:30:21.097606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.097581 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891010-jlbjj"] Oct 31 15:30:21.192528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.192463 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgtp6\" (UniqueName: \"kubernetes.io/projected/bef53331-8c08-41e5-81be-70ed9d941e4b-kube-api-access-wgtp6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:21.270731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.270700 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" Oct 31 15:30:21.271134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.270700 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr" event={"ID":"bef53331-8c08-41e5-81be-70ed9d941e4b","Type":"ContainerDied","Data":"a407987607842e7116f7ba5d5a82d7c742a37340fb95d57ef1bf3e62fa9ec77d"} Oct 31 15:30:21.271134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.270852 4865 scope.go:122] "RemoveContainer" containerID="78d35b100c34accfd86723f72434467847a25794192c6cd429510bfdbd47c0ec" Oct 31 15:30:21.273369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.273349 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" Oct 31 15:30:21.273513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.273351 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2" event={"ID":"76d82b74-7865-4772-be6c-38b4d7a2660b","Type":"ContainerDied","Data":"f4751bed284bb69d5d24b507d02debe48e8bf229e4148b003d5243a0b672c2f6"} Oct 31 15:30:21.274918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.274900 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" Oct 31 15:30:21.275054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.274944 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn" event={"ID":"69cdc270-e7df-4e32-b486-78063e4d3de5","Type":"ContainerDied","Data":"f67e112424e44f90fb06381959f0039770ae39f9b30f0908b2cf74c29ce5d0ec"} Oct 31 15:30:21.284065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.284047 4865 scope.go:122] "RemoveContainer" containerID="0c31e7249d22eb56e97e7c715b01f1b14c0e7646647530c62ae73ea2f7eade1e" Oct 31 15:30:21.294313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.294292 4865 scope.go:122] "RemoveContainer" containerID="13460b793d1366032be6ee7467bfd2efb14d31ae5891967141673e775dbdae21" Oct 31 15:30:21.474736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.474705 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:21.496340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.496318 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-maas-api-service-ca\") pod \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " Oct 31 15:30:21.496508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.496457 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-kube-api-access-p5nc5\" (UniqueName: \"kubernetes.io/projected/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-kube-api-access-p5nc5\") pod \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\" (UID: \"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd\") " Oct 31 15:30:21.496688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.496668 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-maas-api-service-ca" pod "d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" (UID: "d8fedf9b-eb43-4a28-9796-8b5545ca2ddd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:30:21.497024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.497010 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:21.498725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.498701 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-kube-api-access-p5nc5" pod "d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" (UID: "d8fedf9b-eb43-4a28-9796-8b5545ca2ddd"). InnerVolumeSpecName "kube-api-access-p5nc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:30:21.549275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.549251 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2"] Oct 31 15:30:21.553141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.553119 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891010-hdlp2"] Oct 31 15:30:21.598161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.598136 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5nc5\" (UniqueName: \"kubernetes.io/projected/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd-kube-api-access-p5nc5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:30:21.663441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.663373 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr"] Oct 31 15:30:21.668555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.668532 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5fccd7-29891010-mtxmr"] Oct 31 15:30:21.893372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.893337 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn"] Oct 31 15:30:21.897040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:21.897016 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891010-d8ddn"] Oct 31 15:30:22.292747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.292723 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" Oct 31 15:30:22.292747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.292738 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4" event={"ID":"d8fedf9b-eb43-4a28-9796-8b5545ca2ddd","Type":"ContainerDied","Data":"38ec249005c46efe20f7b52e9a30d173134ce24fb24899138ec949b2d4ec1b30"} Oct 31 15:30:22.293266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.292803 4865 scope.go:122] "RemoveContainer" containerID="424a0eb9b2f22b872a1dbf08b55c48e8380a6f185f3f7afcab7964b8d798eaf6" Oct 31 15:30:22.460882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.460854 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4"] Oct 31 15:30:22.466850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.466826 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891010-lf4t4"] Oct 31 15:30:22.498101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.498070 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" path="/var/lib/kubelet/pods/60b21624-f0ba-4179-827b-7dfb79f51d39/volumes" Oct 31 15:30:22.498576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.498561 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" path="/var/lib/kubelet/pods/621c588b-3dc6-40ae-a1c2-a6ddb214669b/volumes" Oct 31 15:30:22.498946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.498932 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" path="/var/lib/kubelet/pods/69cdc270-e7df-4e32-b486-78063e4d3de5/volumes" Oct 31 15:30:22.499639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.499623 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" path="/var/lib/kubelet/pods/76d82b74-7865-4772-be6c-38b4d7a2660b/volumes" Oct 31 15:30:22.500021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.500006 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" path="/var/lib/kubelet/pods/bef53331-8c08-41e5-81be-70ed9d941e4b/volumes" Oct 31 15:30:22.500389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.500376 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" path="/var/lib/kubelet/pods/c45ae376-b982-439f-8d90-10b0e90974ef/volumes" Oct 31 15:30:22.501035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:22.501020 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" path="/var/lib/kubelet/pods/d8fedf9b-eb43-4a28-9796-8b5545ca2ddd/volumes" Oct 31 15:30:25.593133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:25.593098 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:28.189243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:28.189215 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:30:28.189745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:28.189625 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:30:28.191243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:28.191222 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:30:28.547899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:28.547866 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:30.250798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:30.250753 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:33.049449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:33.049401 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f"] Oct 31 15:30:33.122301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:33.122262 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482fd989-vnr4f"] Oct 31 15:30:33.676868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:33.676832 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:33.812555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:33.812517 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:34.048026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:34.047989 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj"] Oct 31 15:30:34.123846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:34.123805 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7266c873-895pj"] Oct 31 15:30:34.523505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:34.522901 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc275da-82c5-4415-9e02-82fb98f8d9cc" path="/var/lib/kubelet/pods/1dc275da-82c5-4415-9e02-82fb98f8d9cc/volumes" Oct 31 15:30:34.523723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:34.523666 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619165bb-95ff-4afe-aa14-13b399874546" path="/var/lib/kubelet/pods/619165bb-95ff-4afe-aa14-13b399874546/volumes" Oct 31 15:30:35.049939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:35.049843 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:37.688163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:37.688131 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:39.483367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.483331 4865 generic.go:356] "Generic (PLEG): container finished" podID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerID="f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593" exitCode=0 Oct 31 15:30:39.483843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.483407 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" event={"ID":"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d","Type":"ContainerDied","Data":"f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593"} Oct 31 15:30:39.483911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.483896 4865 scope.go:122] "RemoveContainer" containerID="f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593" Oct 31 15:30:39.485702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.485671 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:39.485817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.485745 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:39.485817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.485779 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:30:39.495309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.495292 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:30:39.910050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:39.909971 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:40.505751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:40.505715 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" event={"ID":"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d","Type":"ContainerStarted","Data":"5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980"} Oct 31 15:30:40.579825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:40.579732 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:30:40.581967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:40.581562 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 15:30:40.581967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:40.581635 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 15:30:41.579741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.579709 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 15:30:41.580132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.579766 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 15:30:41.822492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.821897 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq"] Oct 31 15:30:41.822977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.822941 4865 cpu_manager.go:414] "containerMap: removing container" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.822977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.822969 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.822984 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.822992 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823009 4865 cpu_manager.go:414] "containerMap: removing container" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823017 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823030 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823035 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823041 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823045 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823054 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823059 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823065 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823069 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823081 4865 cpu_manager.go:414] "containerMap: removing container" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823085 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823097 4865 cpu_manager.go:414] "containerMap: removing container" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823102 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823123 4865 cpu_manager.go:414] "containerMap: removing container" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823128 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823136 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823140 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823148 4865 cpu_manager.go:414] "containerMap: removing container" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823152 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823161 4865 cpu_manager.go:414] "containerMap: removing container" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823165 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823173 4865 cpu_manager.go:414] "containerMap: removing container" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823178 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823183 4865 cpu_manager.go:414] "containerMap: removing container" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823187 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823194 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823198 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:30:41.823185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823208 4865 cpu_manager.go:414] "containerMap: removing container" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823212 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823220 4865 cpu_manager.go:414] "containerMap: removing container" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823224 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823234 4865 cpu_manager.go:414] "containerMap: removing container" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823238 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823249 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823254 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823261 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823265 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823383 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823392 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823397 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823402 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fedf9b-eb43-4a28-9796-8b5545ca2ddd" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823409 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823417 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823426 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823431 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823437 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823443 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823447 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823455 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823461 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823487 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823492 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823497 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823504 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823509 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823514 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823521 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823526 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:30:41.824879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.823532 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:30:41.827461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.827439 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.832179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.832123 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-model-cache\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.832179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.832157 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crgr\" (UniqueName: \"kubernetes.io/projected/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-kube-api-access-6crgr\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.832390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.832363 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tmp-dir\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.832434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.832420 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-home\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.832627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.832608 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tls-certs\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.832781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.832677 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-dshm\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.858267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.858235 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq"] Oct 31 15:30:41.879946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.879922 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:41.917801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.917770 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"openshift-service-ca.crt\"" Oct 31 15:30:41.918664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.918635 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"default-dockercfg-shqgp\"" Oct 31 15:30:41.918817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.918695 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"kube-root-ca.crt\"" Oct 31 15:30:41.918817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.918730 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"auth-test-model-531c3c-kserve-self-signed-certs\"" Oct 31 15:30:41.937491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.935431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tls-certs\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.937491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.935536 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-dshm\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.937491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.935704 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-model-cache\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.937491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.935736 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crgr\" (UniqueName: \"kubernetes.io/projected/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-kube-api-access-6crgr\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.937491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.935888 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tmp-dir\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.952126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.945369 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-home\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.952126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.946280 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tmp-dir\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.952126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.944887 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-home\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.952126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.949197 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-model-cache\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.959677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.958238 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tls-certs\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:41.959677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:41.959101 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-dshm\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:42.018557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:42.018527 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crgr\" (UniqueName: \"kubernetes.io/projected/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-kube-api-access-6crgr\") pod \"auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:42.143053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:42.142966 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:42.549647 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:30:42.549616 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc4bcff4_fa4d_4420_aca1_748ff5e8181c.slice/crio-9024796fa53be4f3078dbce64091377981141c5076eb6781410417631f049f0a WatchSource:0}: Error finding container 9024796fa53be4f3078dbce64091377981141c5076eb6781410417631f049f0a: Status 404 returned error can't find the container with id 9024796fa53be4f3078dbce64091377981141c5076eb6781410417631f049f0a Oct 31 15:30:42.550377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:42.550352 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq"] Oct 31 15:30:42.579709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:42.579679 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 15:30:42.579809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:42.579723 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 15:30:43.542556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:43.542524 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" event={"ID":"bc4bcff4-fa4d-4420-aca1-748ff5e8181c","Type":"ContainerStarted","Data":"e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5"} Oct 31 15:30:43.542757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:43.542566 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" event={"ID":"bc4bcff4-fa4d-4420-aca1-748ff5e8181c","Type":"ContainerStarted","Data":"9024796fa53be4f3078dbce64091377981141c5076eb6781410417631f049f0a"} Oct 31 15:30:43.579276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:43.579241 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 15:30:43.579432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:43.579291 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 15:30:43.676963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:43.676911 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" podStartSLOduration=2.676897361 podStartE2EDuration="2.676897361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:43.674625989 +0000 UTC m=+2689.294177399" watchObservedRunningTime="2026-10-31 15:30:43.676897361 +0000 UTC m=+2689.296448800" Oct 31 15:30:44.143188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:44.143159 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:44.556962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:44.556909 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:44.579510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:44.579455 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 15:30:44.579658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:44.579526 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 15:30:44.719940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:44.719906 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:44.967063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:44.967027 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:45.579588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:45.579553 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 15:30:45.579755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:45.579617 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 15:30:46.584190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:46.584150 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:30:46.584715 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:46.584657 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:30:46.585346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:46.585325 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:30:47.491624 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:30:47.491588 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:30:49.055464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:49.055422 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq"] Oct 31 15:30:49.104117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:49.104081 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d460577-mdxcq"] Oct 31 15:30:49.497226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:49.497192 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:50.499343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:50.499308 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff" path="/var/lib/kubelet/pods/6ac7dda7-cf8c-4726-bc6f-ab41473cb0ff/volumes" Oct 31 15:30:52.143336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:52.143300 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:52.156792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:52.156767 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:52.631689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:52.631657 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:30:54.223602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:54.223561 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:55.940811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:55.940779 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:30:55.949770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:55.949749 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:30:57.354850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:57.354805 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:57.474177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:57.474122 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:58.362884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.362846 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:30:58.373304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.373277 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:30:58.414624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.414594 4865 scope.go:122] "RemoveContainer" containerID="8c46a3b6fe7cf0a2db5681d250dd43eb45e7177f116be3f900f427e0190056f9" Oct 31 15:30:58.425599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.425580 4865 scope.go:122] "RemoveContainer" containerID="948f58e116ec91c94af4462ac6df9e2f064fa62b42ee54286c95cacce19765a0" Oct 31 15:30:58.435600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.435581 4865 scope.go:122] "RemoveContainer" containerID="033b7eda6ef5153b18cdd8e28386e73ab5e66e716fbd1c5c343e78badb229cbf" Oct 31 15:30:58.445523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.445505 4865 scope.go:122] "RemoveContainer" containerID="79360ba0a83df46bba777778693e69a01effaa17ee35cb6a413dfb21a106a1b6" Oct 31 15:30:58.462027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.461996 4865 scope.go:122] "RemoveContainer" containerID="d229f63e55d558b3e3a62a57708e4e27ec8346c079140309c133651ca7e8e67c" Oct 31 15:30:58.476147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.476117 4865 scope.go:122] "RemoveContainer" containerID="21f6c9e4bda8997581cd2bd0b583a9af24766fb4df14fb0568cf6370b71142b8" Oct 31 15:30:58.489641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.489608 4865 scope.go:122] "RemoveContainer" containerID="243c155c737486cc80c7ccf859777b97881adc012461708fc140f15793899b56" Oct 31 15:30:58.995588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:58.995552 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:30:59.056691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:59.056632 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq"] Oct 31 15:30:59.104651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:30:59.104602 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00d2ab5-sfzlq"] Oct 31 15:31:00.323542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:00.323513 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:00.510168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:00.509976 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56877668-57fb-43d1-b731-96593a6c9b6e" path="/var/lib/kubelet/pods/56877668-57fb-43d1-b731-96593a6c9b6e/volumes" Oct 31 15:31:00.660775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:00.660692 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:31:01.788143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:01.788113 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:02.206556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:02.204003 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:04.217495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:04.216016 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:08.915350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:08.915314 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:11.253622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.252930 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": read tcp 10.132.0.2:59642->10.132.1.13:15021: read: connection reset by peer" start-of-body= Oct 31 15:31:11.253622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.253013 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": read tcp 10.132.0.2:59642->10.132.1.13:15021: read: connection reset by peer" Oct 31 15:31:11.803003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.802968 4865 generic.go:356] "Generic (PLEG): container finished" podID="1195e70e-0cf4-433c-884d-9c99833cef37" containerID="66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf" exitCode=0 Oct 31 15:31:11.803233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.803041 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" event={"ID":"1195e70e-0cf4-433c-884d-9c99833cef37","Type":"ContainerDied","Data":"66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf"} Oct 31 15:31:11.803547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.803528 4865 scope.go:122] "RemoveContainer" containerID="66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf" Oct 31 15:31:11.805110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.805082 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:11.805183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.805149 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:11.805226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:11.805197 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:12.054188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:12.054054 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c"] Oct 31 15:31:12.060075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:12.060040 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56716f2b-ltl4c"] Oct 31 15:31:12.519721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:12.519680 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f54bb4-0364-451b-804e-fb43949a56e8" path="/var/lib/kubelet/pods/49f54bb4-0364-451b-804e-fb43949a56e8/volumes" Oct 31 15:31:12.825162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:12.825070 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" event={"ID":"1195e70e-0cf4-433c-884d-9c99833cef37","Type":"ContainerStarted","Data":"9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57"} Oct 31 15:31:13.052426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:13.052394 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn"] Oct 31 15:31:13.055451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:13.055428 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8d4596-lhjrn"] Oct 31 15:31:13.316494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:13.312300 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:13.476538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:13.476504 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:31:13.476749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:13.476680 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 15:31:13.476749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:13.476732 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 15:31:13.921944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:13.921903 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:14.476599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:14.476566 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 15:31:14.476785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:14.476615 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 15:31:14.518899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:14.518866 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9acae8-1773-4465-b14c-d6e9ce158970" path="/var/lib/kubelet/pods/8f9acae8-1773-4465-b14c-d6e9ce158970/volumes" Oct 31 15:31:15.476576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:15.476544 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 15:31:15.476955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:15.476625 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 15:31:16.477178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:16.477143 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 15:31:16.477607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:16.477196 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 15:31:17.476481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:17.476441 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 15:31:17.476637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:17.476525 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 15:31:17.490212 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:31:17.490180 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:31:18.481126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:18.481097 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:31:18.481585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:18.481572 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:31:18.482364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:18.482344 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:31:18.681005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:18.680974 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:18.840825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:18.840739 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:18.965429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:18.965390 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:23.590681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:23.590640 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:25.046643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:25.046607 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg"] Oct 31 15:31:25.097311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:25.097282 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc63d17-rkfzg"] Oct 31 15:31:26.497831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:26.497801 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3887ea21-4360-4fb9-93dc-d634b6b29b2a" path="/var/lib/kubelet/pods/3887ea21-4360-4fb9-93dc-d634b6b29b2a/volumes" Oct 31 15:31:28.233595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:28.233563 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:30.628401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:30.628365 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:31:32.919330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:32.919292 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:34.047267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:34.047232 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs"] Oct 31 15:31:34.121961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:34.121919 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4aa939-62cxs"] Oct 31 15:31:34.521267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:34.521230 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593ce39c-8aff-4e5b-9084-21dec0f18089" path="/var/lib/kubelet/pods/593ce39c-8aff-4e5b-9084-21dec0f18089/volumes" Oct 31 15:31:37.286850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.286820 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt"] Oct 31 15:31:37.287518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287503 4865 cpu_manager.go:414] "containerMap: removing container" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:31:37.287575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287518 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:31:37.287575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287537 4865 cpu_manager.go:414] "containerMap: removing container" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:31:37.287575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287541 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:31:37.287575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287568 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:31:37.287575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287573 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287587 4865 cpu_manager.go:414] "containerMap: removing container" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287591 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621c588b-3dc6-40ae-a1c2-a6ddb214669b" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287603 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287608 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef53331-8c08-41e5-81be-70ed9d941e4b" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287616 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287620 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45ae376-b982-439f-8d90-10b0e90974ef" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287715 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16ecf29-3bde-4477-85bc-59e0291549fa" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287726 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="040913dd-2664-4379-a9c3-c77b61e9c5b0" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287733 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b21624-f0ba-4179-827b-7dfb79f51d39" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287743 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d82b74-7865-4772-be6c-38b4d7a2660b" containerName="cleanup" Oct 31 15:31:37.287750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.287750 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cdc270-e7df-4e32-b486-78063e4d3de5" containerName="cleanup" Oct 31 15:31:37.292440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.292420 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.400784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.400746 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.400964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.400811 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzgd\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-kube-api-access-dgzgd\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.400964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.400913 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.400964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.400951 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.401121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.400996 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ea74b0-0d65-441e-8f8e-1274754236d3-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.401163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.401118 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.401163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.401154 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.401271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.401173 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.401271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.401195 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.492539 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.492506 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt"] Oct 31 15:31:37.502518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502463 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502556 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzgd\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-kube-api-access-dgzgd\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502620 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502657 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502714 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ea74b0-0d65-441e-8f8e-1274754236d3-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502836 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502881 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502908 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.502985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.502944 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.503154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.503056 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.503778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.503757 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.504251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.504209 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.504387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.504344 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.504387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.503825 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ea74b0-0d65-441e-8f8e-1274754236d3-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.507861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.507835 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.507974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.507846 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.561321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.561240 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.561993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.561967 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzgd\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-kube-api-access-dgzgd\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:37.602830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.602802 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:37.607898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:37.607875 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:38.091986 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:31:38.091954 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ea74b0_0d65_441e_8f8e_1274754236d3.slice/crio-f43d1bafc276dc99f7faea0a0dfb24a4c1e99730ac1d5d904753194883be1aa2 WatchSource:0}: Error finding container f43d1bafc276dc99f7faea0a0dfb24a4c1e99730ac1d5d904753194883be1aa2: Status 404 returned error can't find the container with id f43d1bafc276dc99f7faea0a0dfb24a4c1e99730ac1d5d904753194883be1aa2 Oct 31 15:31:38.092110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:38.091978 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt"] Oct 31 15:31:38.095009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:38.094973 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:38.095113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:38.095041 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:38.095113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:38.095072 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:39.081328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:39.081290 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" event={"ID":"d3ea74b0-0d65-441e-8f8e-1274754236d3","Type":"ContainerStarted","Data":"7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee"} Oct 31 15:31:39.082006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:39.081360 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" event={"ID":"d3ea74b0-0d65-441e-8f8e-1274754236d3","Type":"ContainerStarted","Data":"f43d1bafc276dc99f7faea0a0dfb24a4c1e99730ac1d5d904753194883be1aa2"} Oct 31 15:31:39.251410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:39.251348 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" podStartSLOduration=2.251328848 podStartE2EDuration="2.251328848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:31:39.249936899 +0000 UTC m=+2744.869488435" watchObservedRunningTime="2026-10-31 15:31:39.251328848 +0000 UTC m=+2744.870880260" Oct 31 15:31:39.612958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:39.612923 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:39.621500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:39.620643 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 15:31:39.621500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:39.620700 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 15:31:40.588665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:40.586194 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": read tcp 10.132.0.2:44494->10.132.1.23:15021: read: connection reset by peer" start-of-body= Oct 31 15:31:40.588665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:40.586263 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": read tcp 10.132.0.2:44494->10.132.1.23:15021: read: connection reset by peer" Oct 31 15:31:40.618697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:40.614813 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 15:31:40.618697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:40.614885 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 15:31:41.101215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.101178 4865 generic.go:356] "Generic (PLEG): container finished" podID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerID="4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a" exitCode=0 Oct 31 15:31:41.102547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.101503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" event={"ID":"785a65e1-79df-46fd-94dc-f8118b3b2f73","Type":"ContainerDied","Data":"4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a"} Oct 31 15:31:41.103146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.103128 4865 scope.go:122] "RemoveContainer" containerID="4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a" Oct 31 15:31:41.106120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.106077 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:41.106344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.106317 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:41.106517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.106497 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:31:41.618495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.609802 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 15:31:41.618495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:41.609863 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 15:31:42.155151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:42.151710 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" event={"ID":"785a65e1-79df-46fd-94dc-f8118b3b2f73","Type":"ContainerStarted","Data":"a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8"} Oct 31 15:31:42.442674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:42.441967 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:42.621495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:42.618034 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 15:31:42.621495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:42.618115 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 15:31:43.090051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:43.090015 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:31:43.090319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:43.090290 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 15:31:43.090454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:43.090343 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 15:31:43.613496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:43.609898 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 15:31:43.613496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:43.609974 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 15:31:44.090638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:44.090607 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 15:31:44.091040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:44.090658 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 15:31:44.623439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:44.623409 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:44.623905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:44.623882 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:44.624592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:44.624575 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:31:45.090524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:45.090492 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 15:31:45.090694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:45.090545 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 15:31:46.090187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.090149 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 15:31:46.090394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.090217 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 15:31:46.444763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.444684 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w"] Oct 31 15:31:46.449288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.449268 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.482973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.482923 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-dshm\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.483171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.482981 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-tmp-dir\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.483171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.483088 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-model-cache\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.483171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.483150 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xx6\" (UniqueName: \"kubernetes.io/projected/f94a13b8-0e82-44da-80fd-655838216548-kube-api-access-w2xx6\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.483363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.483198 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-home\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.483363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.483348 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a13b8-0e82-44da-80fd-655838216548-tls-certs\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.533238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.533199 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w"] Oct 31 15:31:46.537746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.537715 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:46.575081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.575049 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"openshift-service-ca.crt\"" Oct 31 15:31:46.575306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.575184 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"kube-root-ca.crt\"" Oct 31 15:31:46.575391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.575337 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"default-dockercfg-flk5h\"" Oct 31 15:31:46.575442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.575387 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"auth-test-model-e72453-kserve-self-signed-certs\"" Oct 31 15:31:46.661124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.661083 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-home\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.661325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.661307 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a13b8-0e82-44da-80fd-655838216548-tls-certs\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.661576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.661559 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-dshm\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.661659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.661596 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-tmp-dir\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.661659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.661623 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-model-cache\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.661741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.661691 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xx6\" (UniqueName: \"kubernetes.io/projected/f94a13b8-0e82-44da-80fd-655838216548-kube-api-access-w2xx6\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.662176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.662149 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-home\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.662669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.662645 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-tmp-dir\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.662907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.662874 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-model-cache\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.665437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.665409 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-dshm\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.666021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.666002 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a13b8-0e82-44da-80fd-655838216548-tls-certs\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.722594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.722326 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xx6\" (UniqueName: \"kubernetes.io/projected/f94a13b8-0e82-44da-80fd-655838216548-kube-api-access-w2xx6\") pod \"auth-test-model-e72453-kserve-77c98db4fc-dtv7w\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:46.762889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:46.762855 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:47.090821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:47.090745 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 31 15:31:47.090821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:47.090797 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 31 15:31:47.135052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:47.135014 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:47.181352 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:47.181309 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w"] Oct 31 15:31:47.495031 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:31:47.494994 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:31:48.048570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.048533 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd"] Oct 31 15:31:48.094856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.094825 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:31:48.095173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.095152 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:31:48.096048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.096030 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:31:48.099546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.099518 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9a0904-2fhmd"] Oct 31 15:31:48.245825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.245786 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" event={"ID":"f94a13b8-0e82-44da-80fd-655838216548","Type":"ContainerStarted","Data":"371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4"} Oct 31 15:31:48.245825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.245839 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" event={"ID":"f94a13b8-0e82-44da-80fd-655838216548","Type":"ContainerStarted","Data":"07eca2301b7a8447b9727c305c0f6b16c92141ac0c5b5778fa7d740d3642a016"} Oct 31 15:31:48.454412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.454275 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" podStartSLOduration=2.454257744 podStartE2EDuration="2.454257744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:31:48.453515415 +0000 UTC m=+2754.073066823" watchObservedRunningTime="2026-10-31 15:31:48.454257744 +0000 UTC m=+2754.073809156" Oct 31 15:31:48.498243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.498204 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6890c1a1-c2cf-4eed-a25f-6807c65fb9c4" path="/var/lib/kubelet/pods/6890c1a1-c2cf-4eed-a25f-6807c65fb9c4/volumes" Oct 31 15:31:48.763255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:48.763217 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:51.884805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:51.884775 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:56.573540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:56.573201 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:31:56.763883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:56.763840 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:56.777103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:56.777071 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:57.330545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:57.330510 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:31:58.610706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:58.610668 4865 scope.go:122] "RemoveContainer" containerID="d2aacda5778b719ca596b198994041102c6505f4cbed9b2051a93eaed5795f89" Oct 31 15:31:58.621380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:58.621353 4865 scope.go:122] "RemoveContainer" containerID="56c5dbb525a527c2110422666ada5d1940bd861798d74a37b43b693e293005de" Oct 31 15:31:58.631040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:58.631021 4865 scope.go:122] "RemoveContainer" containerID="be7089ebfe940f7149db79724d26162600aca93788551efef30e6ac9195c58d3" Oct 31 15:31:58.641992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:58.641963 4865 scope.go:122] "RemoveContainer" containerID="26434c9969978554ce2f0c6f91e89cc1f18e7bf09b124d5362fc7d5b7d479cf5" Oct 31 15:31:58.651842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:31:58.651824 4865 scope.go:122] "RemoveContainer" containerID="38686a550eb5373babf153ef8be022c25b51e03a4511e487c875d02a6ff67ae5" Oct 31 15:32:00.686221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:00.684287 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:01.304491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:01.303751 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:02.183764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:02.183729 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:04.730895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:04.730862 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:06.251681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:06.251650 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:06.889423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:06.889385 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:09.261142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.261106 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64"] Oct 31 15:32:09.270351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.270315 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.329422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.329389 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64"] Oct 31 15:32:09.362890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.362856 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:09.374798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.374772 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-home\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.374929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.374909 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tls-certs\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.374990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.374959 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tmp-dir\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.374990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.374983 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-model-cache\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.375171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.375152 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-dshm\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.375243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.375221 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtvc\" (UniqueName: \"kubernetes.io/projected/5f595fb7-0f07-4e54-97ad-e61c8d71a874-kube-api-access-9wtvc\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.375558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.375540 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-21111f\"/\"default-dockercfg-45dbj\"" Oct 31 15:32:09.375558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.375551 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-21111f\"/\"openshift-service-ca.crt\"" Oct 31 15:32:09.375666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.375650 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-21111f\"/\"kube-root-ca.crt\"" Oct 31 15:32:09.375754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.375744 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-21111f\"/\"auth-test-model-21111f-kserve-self-signed-certs\"" Oct 31 15:32:09.476601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.476565 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-home\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.476812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.476723 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tls-certs\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.476812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.476797 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tmp-dir\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.476951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.476828 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-model-cache\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.477022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.476988 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-dshm\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.477089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.477051 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtvc\" (UniqueName: \"kubernetes.io/projected/5f595fb7-0f07-4e54-97ad-e61c8d71a874-kube-api-access-9wtvc\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.477143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.477084 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-home\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.477199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.477137 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tmp-dir\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.477262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.477231 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-model-cache\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.479524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.479500 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-dshm\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.479664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.479582 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tls-certs\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.538622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.538555 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtvc\" (UniqueName: \"kubernetes.io/projected/5f595fb7-0f07-4e54-97ad-e61c8d71a874-kube-api-access-9wtvc\") pod \"auth-test-model-21111f-kserve-779968577c-swb64\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.585162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.585136 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:09.967377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:09.967346 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64"] Oct 31 15:32:09.967829 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:32:09.967799 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f595fb7_0f07_4e54_97ad_e61c8d71a874.slice/crio-e967b068cd9e1bf011300fa3ff44783dd77f9780b84aa167b8c1ea6b8b0cc1fb WatchSource:0}: Error finding container e967b068cd9e1bf011300fa3ff44783dd77f9780b84aa167b8c1ea6b8b0cc1fb: Status 404 returned error can't find the container with id e967b068cd9e1bf011300fa3ff44783dd77f9780b84aa167b8c1ea6b8b0cc1fb Oct 31 15:32:10.438277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:10.438172 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" event={"ID":"5f595fb7-0f07-4e54-97ad-e61c8d71a874","Type":"ContainerStarted","Data":"d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a"} Oct 31 15:32:10.438277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:10.438221 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" event={"ID":"5f595fb7-0f07-4e54-97ad-e61c8d71a874","Type":"ContainerStarted","Data":"e967b068cd9e1bf011300fa3ff44783dd77f9780b84aa167b8c1ea6b8b0cc1fb"} Oct 31 15:32:10.555671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:10.555611 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" podStartSLOduration=1.555595405 podStartE2EDuration="1.555595405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:32:10.553646992 +0000 UTC m=+2776.173198402" watchObservedRunningTime="2026-10-31 15:32:10.555595405 +0000 UTC m=+2776.175146816" Oct 31 15:32:10.586097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:10.586060 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:10.892553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:10.892521 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:12.680974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:12.680937 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:13.865501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:13.865436 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw"] Oct 31 15:32:13.976376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:13.976341 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g"] Oct 31 15:32:14.210278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:14.210244 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:14.383637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:14.383555 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:16.092314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:16.092283 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:17.491246 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:17.491209 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:32:18.472198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:18.472165 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:18.866815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:18.866697 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" podUID="14afe227-bfaf-41f9-8d70-e537d851dd57" containerName="payload-processing" containerID="cri-o://8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9" gracePeriod=25 Oct 31 15:32:18.976999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:18.976936 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" podUID="47e2f292-95c5-4e25-9e3c-c64ae7cd0891" containerName="payload-pre-processing" containerID="cri-o://585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1" gracePeriod=25 Oct 31 15:32:19.173401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.173369 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:32:19.233339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.233292 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14afe227-bfaf-41f9-8d70-e537d851dd57-kube-api-access-7nsgr\" (UniqueName: \"kubernetes.io/projected/14afe227-bfaf-41f9-8d70-e537d851dd57-kube-api-access-7nsgr\") pod \"14afe227-bfaf-41f9-8d70-e537d851dd57\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " Oct 31 15:32:19.233597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.233541 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14afe227-bfaf-41f9-8d70-e537d851dd57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14afe227-bfaf-41f9-8d70-e537d851dd57-plugins-config-volume\") pod \"14afe227-bfaf-41f9-8d70-e537d851dd57\" (UID: \"14afe227-bfaf-41f9-8d70-e537d851dd57\") " Oct 31 15:32:19.233939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.233909 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14afe227-bfaf-41f9-8d70-e537d851dd57-plugins-config-volume" pod "14afe227-bfaf-41f9-8d70-e537d851dd57" (UID: "14afe227-bfaf-41f9-8d70-e537d851dd57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:19.234126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.234108 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14afe227-bfaf-41f9-8d70-e537d851dd57-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:19.236938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.236909 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14afe227-bfaf-41f9-8d70-e537d851dd57-kube-api-access-7nsgr" pod "14afe227-bfaf-41f9-8d70-e537d851dd57" (UID: "14afe227-bfaf-41f9-8d70-e537d851dd57"). InnerVolumeSpecName "kube-api-access-7nsgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:19.273183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.273158 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:32:19.335344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.335307 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-kube-api-access-wwfvk\" (UniqueName: \"kubernetes.io/projected/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-kube-api-access-wwfvk\") pod \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " Oct 31 15:32:19.335550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.335518 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-plugins-config-volume\") pod \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\" (UID: \"47e2f292-95c5-4e25-9e3c-c64ae7cd0891\") " Oct 31 15:32:19.336262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.335982 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-plugins-config-volume" pod "47e2f292-95c5-4e25-9e3c-c64ae7cd0891" (UID: "47e2f292-95c5-4e25-9e3c-c64ae7cd0891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:19.336262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.336214 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:19.336262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.336238 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nsgr\" (UniqueName: \"kubernetes.io/projected/14afe227-bfaf-41f9-8d70-e537d851dd57-kube-api-access-7nsgr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:19.337812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.337787 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-kube-api-access-wwfvk" pod "47e2f292-95c5-4e25-9e3c-c64ae7cd0891" (UID: "47e2f292-95c5-4e25-9e3c-c64ae7cd0891"). InnerVolumeSpecName "kube-api-access-wwfvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:19.437848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.437766 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfvk\" (UniqueName: \"kubernetes.io/projected/47e2f292-95c5-4e25-9e3c-c64ae7cd0891-kube-api-access-wwfvk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:19.511482 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.511429 4865 generic.go:356] "Generic (PLEG): container finished" podID="47e2f292-95c5-4e25-9e3c-c64ae7cd0891" containerID="585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1" exitCode=0 Oct 31 15:32:19.511665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.511498 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" Oct 31 15:32:19.511665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.511512 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" event={"ID":"47e2f292-95c5-4e25-9e3c-c64ae7cd0891","Type":"ContainerDied","Data":"585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1"} Oct 31 15:32:19.511665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.511569 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g" event={"ID":"47e2f292-95c5-4e25-9e3c-c64ae7cd0891","Type":"ContainerDied","Data":"04db2a13dacb6821422d141f7ff95c4367198aaf7a599d6bf3f1dd1bdfb70358"} Oct 31 15:32:19.511665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.511591 4865 scope.go:122] "RemoveContainer" containerID="585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1" Oct 31 15:32:19.513192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.513169 4865 generic.go:356] "Generic (PLEG): container finished" podID="14afe227-bfaf-41f9-8d70-e537d851dd57" containerID="8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9" exitCode=0 Oct 31 15:32:19.513310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.513229 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" Oct 31 15:32:19.513310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.513235 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" event={"ID":"14afe227-bfaf-41f9-8d70-e537d851dd57","Type":"ContainerDied","Data":"8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9"} Oct 31 15:32:19.513310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.513278 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw" event={"ID":"14afe227-bfaf-41f9-8d70-e537d851dd57","Type":"ContainerDied","Data":"e34aebb59bb19e5295eb97254ee8b3fb0202ca9d128484b5ec2d2ccb33522824"} Oct 31 15:32:19.525098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.525078 4865 scope.go:122] "RemoveContainer" containerID="585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1" Oct 31 15:32:19.525431 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:19.525401 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1\": container with ID starting with 585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1 not found: ID does not exist" containerID="585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1" Oct 31 15:32:19.525535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.525447 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1"} err="failed to get container status \"585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1\": rpc error: code = NotFound desc = could not find container \"585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1\": container with ID starting with 585c9023d174d1e043b6bc75b384e6076c04468ba653271e3eb179d06a3244c1 not found: ID does not exist" Oct 31 15:32:19.525535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.525517 4865 scope.go:122] "RemoveContainer" containerID="8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9" Oct 31 15:32:19.540671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.539994 4865 scope.go:122] "RemoveContainer" containerID="8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9" Oct 31 15:32:19.540751 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:19.540727 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9\": container with ID starting with 8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9 not found: ID does not exist" containerID="8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9" Oct 31 15:32:19.540818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.540760 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9"} err="failed to get container status \"8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9\": rpc error: code = NotFound desc = could not find container \"8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9\": container with ID starting with 8fc332f85d774d70035e34f7163ef26d9b17359bd87bb10be13141de33071bd9 not found: ID does not exist" Oct 31 15:32:19.585723 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.585685 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:19.598796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.598765 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:19.680839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.680805 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g"] Oct 31 15:32:19.684172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.684147 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5fccd7-6bdbfbd5bc-7kp5g"] Oct 31 15:32:19.810615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.810577 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw"] Oct 31 15:32:19.814036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:19.814006 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5fccd7-8645f5887-2bmnw"] Oct 31 15:32:20.498361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:20.498326 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14afe227-bfaf-41f9-8d70-e537d851dd57" path="/var/lib/kubelet/pods/14afe227-bfaf-41f9-8d70-e537d851dd57/volumes" Oct 31 15:32:20.498856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:20.498839 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e2f292-95c5-4e25-9e3c-c64ae7cd0891" path="/var/lib/kubelet/pods/47e2f292-95c5-4e25-9e3c-c64ae7cd0891/volumes" Oct 31 15:32:20.534000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:20.533968 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:32:20.797338 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:20.797246 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:32:20.832636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:20.832598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:22.913404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:22.913365 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc"] Oct 31 15:32:22.913930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:22.913630 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" containerID="cri-o://bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575" gracePeriod=30 Oct 31 15:32:25.509336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:25.509304 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:26.619940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.619907 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6"] Oct 31 15:32:26.620678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.620658 4865 cpu_manager.go:414] "containerMap: removing container" podUID="47e2f292-95c5-4e25-9e3c-c64ae7cd0891" containerName="payload-pre-processing" Oct 31 15:32:26.620678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.620677 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e2f292-95c5-4e25-9e3c-c64ae7cd0891" containerName="payload-pre-processing" Oct 31 15:32:26.620867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.620689 4865 cpu_manager.go:414] "containerMap: removing container" podUID="14afe227-bfaf-41f9-8d70-e537d851dd57" containerName="payload-processing" Oct 31 15:32:26.620867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.620694 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14afe227-bfaf-41f9-8d70-e537d851dd57" containerName="payload-processing" Oct 31 15:32:26.620867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.620810 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e2f292-95c5-4e25-9e3c-c64ae7cd0891" containerName="payload-pre-processing" Oct 31 15:32:26.620867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.620821 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="14afe227-bfaf-41f9-8d70-e537d851dd57" containerName="payload-processing" Oct 31 15:32:26.624342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.624324 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.682382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.682349 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6"] Oct 31 15:32:26.725380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.725344 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:26.726944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.726917 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"openshift-service-ca.crt\"" Oct 31 15:32:26.726944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.726926 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"kube-root-ca.crt\"" Oct 31 15:32:26.727093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.726964 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"default-dockercfg-8kj8z\"" Oct 31 15:32:26.729247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.729229 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"auth-test-model-55b3fb-kserve-self-signed-certs\"" Oct 31 15:32:26.766585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.766531 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdts\" (UniqueName: \"kubernetes.io/projected/b80e12f0-24c8-4b17-8201-1d6d5b259771-kube-api-access-lqdts\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.767743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.767683 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80e12f0-24c8-4b17-8201-1d6d5b259771-tls-certs\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.768075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.767929 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-model-cache\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.768255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.768077 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-dshm\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.768255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.768135 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-home\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.768255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.768167 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-tmp-dir\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.869856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.869820 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80e12f0-24c8-4b17-8201-1d6d5b259771-tls-certs\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.869957 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-model-cache\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.870023 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-dshm\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.870069 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-home\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.870100 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-tmp-dir\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.870154 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdts\" (UniqueName: \"kubernetes.io/projected/b80e12f0-24c8-4b17-8201-1d6d5b259771-kube-api-access-lqdts\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.870413 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-model-cache\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.870485 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-home\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.870557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.870526 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-tmp-dir\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.872400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.872378 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-dshm\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.872615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.872599 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80e12f0-24c8-4b17-8201-1d6d5b259771-tls-certs\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:26.941105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:26.941073 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdts\" (UniqueName: \"kubernetes.io/projected/b80e12f0-24c8-4b17-8201-1d6d5b259771-kube-api-access-lqdts\") pod \"auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:27.237142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:27.237107 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:27.615182 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:32:27.615151 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80e12f0_24c8_4b17_8201_1d6d5b259771.slice/crio-83a21427ec437931e9078ec7e220eb3539f1c22cf784b49e394f61ba33a12d4b WatchSource:0}: Error finding container 83a21427ec437931e9078ec7e220eb3539f1c22cf784b49e394f61ba33a12d4b: Status 404 returned error can't find the container with id 83a21427ec437931e9078ec7e220eb3539f1c22cf784b49e394f61ba33a12d4b Oct 31 15:32:27.615903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:27.615880 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6"] Oct 31 15:32:28.200873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.200811 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:32:28.288668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.288631 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-kube-api-access-nk65k\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-kube-api-access-nk65k\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.288872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.288783 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-envoy\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.288872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.288823 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-credential-socket\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.289021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.288905 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-certs\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.289021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.288982 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-socket\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.289145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289104 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-data\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.289145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289135 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-podinfo\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.289266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289161 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istiod-ca-cert\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.289266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289190 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-token\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-token\") pod \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\" (UID: \"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36\") " Oct 31 15:32:28.289266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289199 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-certs" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:28.289266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289217 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-credential-socket" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:28.289266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289239 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-socket" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:28.289759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289458 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-data" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:28.289759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.289596 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istiod-ca-cert" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:28.290201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.290176 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.290321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.290205 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.290321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.290221 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.290321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.290244 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.290321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.290258 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.291223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.291203 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-envoy" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:28.291373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.291357 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-kube-api-access-nk65k" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "kube-api-access-nk65k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:28.291454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.291437 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-podinfo" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:28.291579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.291561 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-token" pod "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" (UID: "bd7c7719-b0dc-4ccc-a46c-d295a38bdf36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:28.392354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.392313 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk65k\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-kube-api-access-nk65k\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.392558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.392361 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.392558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.392377 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.392558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.392391 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:28.590807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.590763 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" event={"ID":"b80e12f0-24c8-4b17-8201-1d6d5b259771","Type":"ContainerStarted","Data":"d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4"} Oct 31 15:32:28.590975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.590814 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" event={"ID":"b80e12f0-24c8-4b17-8201-1d6d5b259771","Type":"ContainerStarted","Data":"83a21427ec437931e9078ec7e220eb3539f1c22cf784b49e394f61ba33a12d4b"} Oct 31 15:32:28.592369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.592343 4865 generic.go:356] "Generic (PLEG): container finished" podID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerID="bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575" exitCode=0 Oct 31 15:32:28.592560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.592387 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" Oct 31 15:32:28.592560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.592425 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" event={"ID":"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36","Type":"ContainerDied","Data":"bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575"} Oct 31 15:32:28.592560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.592484 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc" event={"ID":"bd7c7719-b0dc-4ccc-a46c-d295a38bdf36","Type":"ContainerDied","Data":"47d5e1cdde5b707376fc24ae70c858f49236ab597fb416f03482f25b8765bb0c"} Oct 31 15:32:28.592560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.592514 4865 scope.go:122] "RemoveContainer" containerID="bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575" Oct 31 15:32:28.603131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.603108 4865 scope.go:122] "RemoveContainer" containerID="bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575" Oct 31 15:32:28.603419 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:28.603391 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575\": container with ID starting with bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575 not found: ID does not exist" containerID="bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575" Oct 31 15:32:28.603545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.603431 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575"} err="failed to get container status \"bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575\": rpc error: code = NotFound desc = could not find container \"bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575\": container with ID starting with bb59c99f37098e9c9bd3dba10f4e78b9413b340abe4137d727720ec322167575 not found: ID does not exist" Oct 31 15:32:28.710294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.710243 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" podStartSLOduration=2.710219384 podStartE2EDuration="2.710219384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:32:28.706587358 +0000 UTC m=+2794.326138768" watchObservedRunningTime="2026-10-31 15:32:28.710219384 +0000 UTC m=+2794.329770797" Oct 31 15:32:28.836505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.834196 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc"] Oct 31 15:32:28.874146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:28.874113 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-6hmfc"] Oct 31 15:32:29.237339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:29.237311 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:30.241852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:30.241817 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:30.499449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:30.499360 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" path="/var/lib/kubelet/pods/bd7c7719-b0dc-4ccc-a46c-d295a38bdf36/volumes" Oct 31 15:32:32.046348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:32.046319 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7"] Oct 31 15:32:32.097215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:32.097187 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01328dd6-k2cv7"] Oct 31 15:32:32.498551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:32.498517 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b7a2e9-7541-4540-81da-90e62cdf2409" path="/var/lib/kubelet/pods/a3b7a2e9-7541-4540-81da-90e62cdf2409/volumes" Oct 31 15:32:34.925944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:34.925908 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:37.237770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:37.237740 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:37.251437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:37.251404 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:37.687448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:37.687364 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:32:38.887647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:38.887571 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:39.614591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:39.614562 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8"] Oct 31 15:32:39.614800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:39.614778 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" containerID="cri-o://7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6" gracePeriod=30 Oct 31 15:32:39.665737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:39.665694 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd"] Oct 31 15:32:39.665973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:39.665941 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" containerID="cri-o://964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798" gracePeriod=30 Oct 31 15:32:39.671069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:39.671040 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78"] Oct 31 15:32:39.671293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:39.671248 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" containerID="cri-o://b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf" gracePeriod=30 Oct 31 15:32:39.910092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:39.910004 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:44.627326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:44.627289 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:44.957188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:44.957159 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:32:45.012463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.012442 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:32:45.016368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.016349 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:32:45.135075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135041 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-certs\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.135266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135107 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-envoy\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.135266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135134 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-data\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.135266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135152 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-certs\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.135266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135173 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-socket\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.135266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135246 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-kube-api-access-pqrqq\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-kube-api-access-pqrqq\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135275 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-podinfo\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135313 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-certs\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135357 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-kube-api-access-wj5rh\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-kube-api-access-wj5rh\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135374 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-certs" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135392 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-envoy\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135439 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-data" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135463 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-credential-socket\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135520 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-socket\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.135624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135577 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-credential-socket\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135643 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-envoy\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135680 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-token\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-token\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135703 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-certs" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135716 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istiod-ca-cert\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135726 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-certs" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135752 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-podinfo\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135808 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istiod-ca-cert\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135841 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-podinfo\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135891 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-kube-api-access-swz2d\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-kube-api-access-swz2d\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135943 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-data\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.135973 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-credential-socket\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.136007 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-credential-socket" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.136096 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-token\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-token\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.136137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.136134 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-data\") pod \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\" (UID: \"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c\") " Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.136181 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-token\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.136213 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-credential-socket" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.136221 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-socket\") pod \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\" (UID: \"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba\") " Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.136307 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istiod-ca-cert\") pod \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\" (UID: \"5c295cf7-ae6a-4797-ac03-87176fa70cc6\") " Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137114 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137139 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137153 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137167 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137181 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137194 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137530 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istiod-ca-cert" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.137728 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-socket" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.138094 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istiod-ca-cert" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.138493 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istiod-ca-cert" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.138668 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-kube-api-access-wj5rh" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "kube-api-access-wj5rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.138736 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-envoy" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.138787 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-envoy" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.139089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.139060 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-socket" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.140163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.139168 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-socket" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.140163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.139489 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-credential-socket" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.140163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.139748 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-data" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.140163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.139755 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-data" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.141389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.141358 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-kube-api-access-swz2d" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "kube-api-access-swz2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:45.141837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.141806 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-podinfo" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:45.141944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.141849 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-envoy" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:45.142104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.142071 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-podinfo" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:45.143724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.143595 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-podinfo" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:45.143724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.143638 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-token" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:45.143724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.143671 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-token" pod "5c295cf7-ae6a-4797-ac03-87176fa70cc6" (UID: "5c295cf7-ae6a-4797-ac03-87176fa70cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:45.143724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.143670 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-kube-api-access-pqrqq" pod "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" (UID: "fbebde03-06ea-43c3-afff-a2cfbb4b4a3c"). InnerVolumeSpecName "kube-api-access-pqrqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:45.143724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.143692 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-token" pod "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" (UID: "7d1a7f85-00e6-4f94-b74b-af9cdd2367ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:45.238847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238809 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.238847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238838 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.238847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238847 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.238847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238856 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238865 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238879 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238891 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238900 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrqq\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-kube-api-access-pqrqq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238910 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238919 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5rh\" (UniqueName: \"kubernetes.io/projected/5c295cf7-ae6a-4797-ac03-87176fa70cc6-kube-api-access-wj5rh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238928 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238936 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238945 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238961 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238969 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238979 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238987 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.238995 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.239003 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.239012 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swz2d\" (UniqueName: \"kubernetes.io/projected/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba-kube-api-access-swz2d\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.239327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.239022 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c295cf7-ae6a-4797-ac03-87176fa70cc6-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:45.745664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.745629 4865 generic.go:356] "Generic (PLEG): container finished" podID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerID="7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6" exitCode=0 Oct 31 15:32:45.746144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.745702 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" Oct 31 15:32:45.746144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.745718 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" event={"ID":"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c","Type":"ContainerDied","Data":"7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6"} Oct 31 15:32:45.746144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.745780 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8" event={"ID":"fbebde03-06ea-43c3-afff-a2cfbb4b4a3c","Type":"ContainerDied","Data":"9e36d80884faedbee4e264f2070e3a0625f1b99be66f02ec4f8ffe909190d7c3"} Oct 31 15:32:45.746144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.745812 4865 scope.go:122] "RemoveContainer" containerID="7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6" Oct 31 15:32:45.747447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.747419 4865 generic.go:356] "Generic (PLEG): container finished" podID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerID="964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798" exitCode=0 Oct 31 15:32:45.747622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.747556 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" Oct 31 15:32:45.747789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.747713 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" event={"ID":"5c295cf7-ae6a-4797-ac03-87176fa70cc6","Type":"ContainerDied","Data":"964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798"} Oct 31 15:32:45.747789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.747760 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" event={"ID":"5c295cf7-ae6a-4797-ac03-87176fa70cc6","Type":"ContainerDied","Data":"be58892761caf67a5fa51c3ce587ad83a279fc2d49b2d41696d783094c32faf3"} Oct 31 15:32:45.749444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.749424 4865 generic.go:356] "Generic (PLEG): container finished" podID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerID="b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf" exitCode=0 Oct 31 15:32:45.749586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.749516 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" event={"ID":"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba","Type":"ContainerDied","Data":"b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf"} Oct 31 15:32:45.749586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.749556 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" event={"ID":"7d1a7f85-00e6-4f94-b74b-af9cdd2367ba","Type":"ContainerDied","Data":"11c8b424e7f3f8feb59ce2e55ef6b2b9a8309c4117c8f059e9155033188704dd"} Oct 31 15:32:45.749682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.749590 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78" Oct 31 15:32:45.759871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.759791 4865 scope.go:122] "RemoveContainer" containerID="7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6" Oct 31 15:32:45.760130 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:45.760106 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6\": container with ID starting with 7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6 not found: ID does not exist" containerID="7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6" Oct 31 15:32:45.760208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.760142 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6"} err="failed to get container status \"7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6\": rpc error: code = NotFound desc = could not find container \"7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6\": container with ID starting with 7f9fe942f6b8772facc31ec3ba3650e56b77f24362dabf8fc58af31db831eec6 not found: ID does not exist" Oct 31 15:32:45.760208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.760163 4865 scope.go:122] "RemoveContainer" containerID="964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798" Oct 31 15:32:45.770740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.770719 4865 scope.go:122] "RemoveContainer" containerID="964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798" Oct 31 15:32:45.771011 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:45.770991 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798\": container with ID starting with 964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798 not found: ID does not exist" containerID="964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798" Oct 31 15:32:45.771097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.771019 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798"} err="failed to get container status \"964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798\": rpc error: code = NotFound desc = could not find container \"964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798\": container with ID starting with 964b1cbc9c7ff2a35b2018b5b7f71924ae6531a4eb0265bf847586e410fd7798 not found: ID does not exist" Oct 31 15:32:45.771097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.771042 4865 scope.go:122] "RemoveContainer" containerID="b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf" Oct 31 15:32:45.780786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.780593 4865 scope.go:122] "RemoveContainer" containerID="b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf" Oct 31 15:32:45.780846 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:45.780829 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf\": container with ID starting with b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf not found: ID does not exist" containerID="b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf" Oct 31 15:32:45.780892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.780853 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf"} err="failed to get container status \"b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf\": rpc error: code = NotFound desc = could not find container \"b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf\": container with ID starting with b745f8de25795b81dfee9052534010906b06aee38c829a3c6daef7dbffbda2cf not found: ID does not exist" Oct 31 15:32:45.914625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.914543 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8"] Oct 31 15:32:45.919359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.919331 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-qsjv8"] Oct 31 15:32:45.934212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.934183 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:32:45.934341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:45.934248 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:32:46.030209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:46.030179 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd"] Oct 31 15:32:46.033880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:46.033856 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-nxbcd"] Oct 31 15:32:46.146238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:46.146200 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78"] Oct 31 15:32:46.149705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:46.149680 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dwv78"] Oct 31 15:32:46.497696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:46.497668 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" path="/var/lib/kubelet/pods/5c295cf7-ae6a-4797-ac03-87176fa70cc6/volumes" Oct 31 15:32:46.498154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:46.498140 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" path="/var/lib/kubelet/pods/7d1a7f85-00e6-4f94-b74b-af9cdd2367ba/volumes" Oct 31 15:32:46.498610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:46.498597 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" path="/var/lib/kubelet/pods/fbebde03-06ea-43c3-afff-a2cfbb4b4a3c/volumes" Oct 31 15:32:47.490990 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:32:47.490956 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:32:49.401189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:49.401155 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:54.189413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:54.189380 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:57.876135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:57.876099 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j"] Oct 31 15:32:57.876692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:57.876299 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" containerID="cri-o://e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0" gracePeriod=30 Oct 31 15:32:58.372846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:58.372811 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:32:58.715360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:58.715329 4865 scope.go:122] "RemoveContainer" containerID="3463c6308d9c09c94bb8e52005a1fcb9971d61f5927021a5b932f520ad52e24f" Oct 31 15:32:58.727097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:58.727074 4865 scope.go:122] "RemoveContainer" containerID="4c5180a94689f5eb01fe378aaafcd83c64c067de4b04ce3e5e77618934aaa07c" Oct 31 15:32:58.736855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:58.736834 4865 scope.go:122] "RemoveContainer" containerID="ab6d807de94a97fdc89a9d6e5d3545984dac08e3384dae81a745325501b46acd" Oct 31 15:32:58.746558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:58.746538 4865 scope.go:122] "RemoveContainer" containerID="18f48cde5f3d375a0ab51162979488e8b561880783587ee89fc527a96996cd2f" Oct 31 15:32:58.756349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:58.756321 4865 scope.go:122] "RemoveContainer" containerID="c1c442b857cbbf656ca384978823ad1071743298075a01c61d53f5e51446a256" Oct 31 15:32:59.986673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:32:59.986637 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:01.025835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:01.025792 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:02.467486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:02.467439 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:03.144287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.144262 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:33:03.302694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.302661 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-credential-socket\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.302907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.302709 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istiod-ca-cert\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.302907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.302741 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-kube-api-access-6s5kn\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-kube-api-access-6s5kn\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.302907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.302798 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-certs\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.302907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.302843 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-podinfo\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.303169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.302947 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-envoy\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.303169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.302997 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-token\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-token\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.303169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.303023 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-credential-socket" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:03.303169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.303070 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-data\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.303169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.303129 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istiod-ca-cert" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:33:03.303448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.303165 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-certs" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:03.303448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.303201 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-socket\") pod \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\" (UID: \"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3\") " Oct 31 15:33:03.303581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.303444 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-data" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:03.303581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.303490 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-socket" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:03.304108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.304085 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.304232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.304113 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.304232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.304129 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.304232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.304142 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.304232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.304157 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.305446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.305419 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-podinfo" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:33:03.305599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.305552 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-envoy" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:03.305647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.305628 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-kube-api-access-6s5kn" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "kube-api-access-6s5kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:03.305801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.305783 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-token" pod "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" (UID: "7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:03.405283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.405244 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.405283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.405276 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.405283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.405285 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.405655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.405295 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s5kn\" (UniqueName: \"kubernetes.io/projected/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3-kube-api-access-6s5kn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:03.896034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.895996 4865 generic.go:356] "Generic (PLEG): container finished" podID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerID="e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0" exitCode=0 Oct 31 15:33:03.896497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.896043 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" Oct 31 15:33:03.896497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.896085 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" event={"ID":"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3","Type":"ContainerDied","Data":"e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0"} Oct 31 15:33:03.896497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.896138 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j" event={"ID":"7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3","Type":"ContainerDied","Data":"723db1673175f6c53b8a2d63253cae997c050841e3cbe347fc99f21d3044728f"} Oct 31 15:33:03.896497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.896164 4865 scope.go:122] "RemoveContainer" containerID="e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0" Oct 31 15:33:03.907780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.907758 4865 scope.go:122] "RemoveContainer" containerID="e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0" Oct 31 15:33:03.908045 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:33:03.908025 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0\": container with ID starting with e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0 not found: ID does not exist" containerID="e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0" Oct 31 15:33:03.908094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:03.908052 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0"} err="failed to get container status \"e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0\": rpc error: code = NotFound desc = could not find container \"e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0\": container with ID starting with e8b5c544e0f9ea639af341bf509a543e659867611778250684716db46fe5a7a0 not found: ID does not exist" Oct 31 15:33:04.117721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:04.117688 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j"] Oct 31 15:33:04.126174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:04.123251 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-cnz8j"] Oct 31 15:33:04.497583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:04.497554 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" path="/var/lib/kubelet/pods/7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3/volumes" Oct 31 15:33:04.696991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:04.696963 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:07.418663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:07.418592 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:07.546688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:07.546660 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:08.995813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:08.995777 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:09.628101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:09.628044 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:09.665108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:09.665069 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv"] Oct 31 15:33:09.665318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:09.665297 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" containerID="cri-o://b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c" gracePeriod=30 Oct 31 15:33:09.763511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:09.759878 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:09.943961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:09.943788 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:12.110335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:12.110303 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:33:14.478064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.478022 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:14.917096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.917057 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:14.928075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.928054 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:33:14.982631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.982599 4865 generic.go:356] "Generic (PLEG): container finished" podID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerID="b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c" exitCode=0 Oct 31 15:33:14.982765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.982643 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" Oct 31 15:33:14.982765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.982658 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" event={"ID":"eb908f7b-125c-4283-a4cd-09e603a2c4d9","Type":"ContainerDied","Data":"b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c"} Oct 31 15:33:14.982765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.982699 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv" event={"ID":"eb908f7b-125c-4283-a4cd-09e603a2c4d9","Type":"ContainerDied","Data":"a68a28dfe5c709b39fec3a88492b7028bc092c1e8ab91207c6d799c44f277371"} Oct 31 15:33:14.982765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.982720 4865 scope.go:122] "RemoveContainer" containerID="b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c" Oct 31 15:33:14.993004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.992986 4865 scope.go:122] "RemoveContainer" containerID="b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f" Oct 31 15:33:14.998734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.998708 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-podinfo\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.998835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.998772 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-token\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.998835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.998831 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-envoy\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.998973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.998867 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-kube-api-access-nfxh5\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-kube-api-access-nfxh5\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.998973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.998946 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-data\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.999094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.998982 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istiod-ca-cert\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.999094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.999027 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-socket\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.999094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.999069 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-certs\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.999269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.999185 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-credential-socket\") pod \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\" (UID: \"eb908f7b-125c-4283-a4cd-09e603a2c4d9\") " Oct 31 15:33:14.999328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.999279 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-data" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:14.999679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.999608 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istiod-ca-cert" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:33:14.999762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.999715 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-socket" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:15.000001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:14.999746 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-certs" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:15.000068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.000010 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.000068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.000031 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.000068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.000047 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.000068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.000061 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.000410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.000370 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-credential-socket" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:15.001256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.001235 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-envoy" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:15.001365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.001321 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-kube-api-access-nfxh5" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "kube-api-access-nfxh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:15.001547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.001529 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-token" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:15.001616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.001543 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-podinfo" pod "eb908f7b-125c-4283-a4cd-09e603a2c4d9" (UID: "eb908f7b-125c-4283-a4cd-09e603a2c4d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:33:15.003753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.003737 4865 scope.go:122] "RemoveContainer" containerID="b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c" Oct 31 15:33:15.004006 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:33:15.003987 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c\": container with ID starting with b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c not found: ID does not exist" containerID="b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c" Oct 31 15:33:15.004070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.004012 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c"} err="failed to get container status \"b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c\": rpc error: code = NotFound desc = could not find container \"b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c\": container with ID starting with b89db66384d3db27fcdac8dcdceea1f6760f61dbb0ff48c844dcc02c9365ca3c not found: ID does not exist" Oct 31 15:33:15.004070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.004029 4865 scope.go:122] "RemoveContainer" containerID="b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f" Oct 31 15:33:15.004281 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:33:15.004265 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f\": container with ID starting with b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f not found: ID does not exist" containerID="b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f" Oct 31 15:33:15.004332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.004286 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f"} err="failed to get container status \"b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f\": rpc error: code = NotFound desc = could not find container \"b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f\": container with ID starting with b1e47eb7a5dfc3b069460ef7f6a8b95b707e6048a96f7292d69d7bf4345c631f not found: ID does not exist" Oct 31 15:33:15.101030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.100970 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.101030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.100994 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.101030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.101005 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.101030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.101015 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxh5\" (UniqueName: \"kubernetes.io/projected/eb908f7b-125c-4283-a4cd-09e603a2c4d9-kube-api-access-nfxh5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.101030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.101024 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f7b-125c-4283-a4cd-09e603a2c4d9-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:15.407027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.406939 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv"] Oct 31 15:33:15.410568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:15.410542 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5fccd7-openshift-default-79f9978bb9-t7hjv"] Oct 31 15:33:16.498790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:16.498761 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" path="/var/lib/kubelet/pods/eb908f7b-125c-4283-a4cd-09e603a2c4d9/volumes" Oct 31 15:33:17.490643 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:33:17.490604 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:33:19.095853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:19.095810 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:20.264589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:20.264556 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq"] Oct 31 15:33:20.265171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:20.264810 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" containerID="cri-o://4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277" gracePeriod=30 Oct 31 15:33:20.493733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:20.493699 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:20.656694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:20.656604 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:20.861501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:20.861442 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:24.236010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:24.235977 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:25.560328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.560297 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:33:25.642523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642490 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-istio-token\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-istio-token\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.642716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642584 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-certs\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.642716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642619 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e1b9e0-f557-43a6-895b-249e01261062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e1b9e0-f557-43a6-895b-249e01261062-istiod-ca-cert\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.642716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642688 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8e1b9e0-f557-43a6-895b-249e01261062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e1b9e0-f557-43a6-895b-249e01261062-istio-podinfo\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.642890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642773 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-credential-socket\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.642890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642846 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-envoy\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.643011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642908 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-data\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.643011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642930 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-certs" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:25.643011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.642985 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-kube-api-access-zvvc8\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-kube-api-access-zvvc8\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.643175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643015 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-socket\") pod \"a8e1b9e0-f557-43a6-895b-249e01261062\" (UID: \"a8e1b9e0-f557-43a6-895b-249e01261062\") " Oct 31 15:33:25.643175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643057 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e1b9e0-f557-43a6-895b-249e01261062-istiod-ca-cert" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:33:25.643287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643194 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-data" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:25.643391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643367 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-credential-socket" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:25.643454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643403 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-socket" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:25.643771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643662 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.643771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643687 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.643771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643702 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.643771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643716 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.643771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.643732 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e1b9e0-f557-43a6-895b-249e01261062-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.645906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.645875 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-kube-api-access-zvvc8" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "kube-api-access-zvvc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:25.645906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.645898 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-istio-token" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:25.646070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.645909 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-envoy" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:25.646117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.646072 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e1b9e0-f557-43a6-895b-249e01261062-istio-podinfo" pod "a8e1b9e0-f557-43a6-895b-249e01261062" (UID: "a8e1b9e0-f557-43a6-895b-249e01261062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:33:25.745847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.745803 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e1b9e0-f557-43a6-895b-249e01261062-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.745847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.745840 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e1b9e0-f557-43a6-895b-249e01261062-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.746115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.745857 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvc8\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-kube-api-access-zvvc8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.746115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:25.745872 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e1b9e0-f557-43a6-895b-249e01261062-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:26.058392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.058359 4865 generic.go:356] "Generic (PLEG): container finished" podID="a8e1b9e0-f557-43a6-895b-249e01261062" containerID="4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277" exitCode=0 Oct 31 15:33:26.058601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.058404 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" Oct 31 15:33:26.058601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.058442 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" event={"ID":"a8e1b9e0-f557-43a6-895b-249e01261062","Type":"ContainerDied","Data":"4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277"} Oct 31 15:33:26.058601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.058570 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq" event={"ID":"a8e1b9e0-f557-43a6-895b-249e01261062","Type":"ContainerDied","Data":"d897fa4855e01a7e51e16fd4de1119f93f274ed06fd349d9d031db02730eee42"} Oct 31 15:33:26.058745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.058602 4865 scope.go:122] "RemoveContainer" containerID="4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277" Oct 31 15:33:26.070170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.070145 4865 scope.go:122] "RemoveContainer" containerID="4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277" Oct 31 15:33:26.070462 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:33:26.070443 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277\": container with ID starting with 4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277 not found: ID does not exist" containerID="4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277" Oct 31 15:33:26.070548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.070487 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277"} err="failed to get container status \"4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277\": rpc error: code = NotFound desc = could not find container \"4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277\": container with ID starting with 4fdc86f78b1816b99eee34fa1d43cd316006bec8b07a5cd51b755f7bf9268277 not found: ID does not exist" Oct 31 15:33:26.227004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.226970 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq"] Oct 31 15:33:26.278049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.278021 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-xpnjq"] Oct 31 15:33:26.399734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.399662 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:26.497226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:26.497194 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" path="/var/lib/kubelet/pods/a8e1b9e0-f557-43a6-895b-249e01261062/volumes" Oct 31 15:33:29.012753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:29.012724 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:33.665216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:33.665176 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:38.459224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:38.459191 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:43.251371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.251333 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:43.315899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.315867 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr"] Oct 31 15:33:43.316622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316601 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" Oct 31 15:33:43.316622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316623 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316635 4865 cpu_manager.go:414] "containerMap: removing container" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316642 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316670 4865 cpu_manager.go:414] "containerMap: removing container" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316677 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316697 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316704 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316726 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316734 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316760 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316767 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316787 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316796 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316817 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" Oct 31 15:33:43.316836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316825 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316982 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e1b9e0-f557-43a6-895b-249e01261062" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.316996 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c295cf7-ae6a-4797-ac03-87176fa70cc6" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.317009 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd730bc-0df8-4ecb-8c3a-2272f6aa1be3" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.317022 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.317035 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb908f7b-125c-4283-a4cd-09e603a2c4d9" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.317049 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1a7f85-00e6-4f94-b74b-af9cdd2367ba" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.317061 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbebde03-06ea-43c3-afff-a2cfbb4b4a3c" containerName="istio-proxy" Oct 31 15:33:43.317585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.317073 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7c7719-b0dc-4ccc-a46c-d295a38bdf36" containerName="istio-proxy" Oct 31 15:33:43.326365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.326337 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.374132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.374098 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr"] Oct 31 15:33:43.382893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.382863 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfbb5\" (UniqueName: \"kubernetes.io/projected/d9a0839d-f560-4a73-a294-60e3aa5f94f6-kube-api-access-gfbb5\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.383060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.382915 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-home\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.383145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.383063 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-model-cache\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.383209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.383183 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-dshm\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.383370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.383349 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tmp-dir\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.383426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.383400 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tls-certs\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.404402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.404375 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:43.421105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.421073 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2ad95a\"/\"kube-root-ca.crt\"" Oct 31 15:33:43.421105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.421100 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2ad95a\"/\"default-dockercfg-krfc7\"" Oct 31 15:33:43.431316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.431295 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2ad95a\"/\"openshift-service-ca.crt\"" Oct 31 15:33:43.431389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.431318 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2ad95a\"/\"autores-model-2ad95a-kserve-self-signed-certs\"" Oct 31 15:33:43.485430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.485399 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-dshm\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.485628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.485586 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tmp-dir\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.485680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.485631 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tls-certs\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.485758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.485741 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfbb5\" (UniqueName: \"kubernetes.io/projected/d9a0839d-f560-4a73-a294-60e3aa5f94f6-kube-api-access-gfbb5\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.485831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.485817 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-home\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.485876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.485865 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-model-cache\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.486003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.485980 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tmp-dir\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.486223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.486164 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-home\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.486223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.486193 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-model-cache\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.487916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.487894 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-dshm\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.488384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.488312 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tls-certs\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.544887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.544807 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfbb5\" (UniqueName: \"kubernetes.io/projected/d9a0839d-f560-4a73-a294-60e3aa5f94f6-kube-api-access-gfbb5\") pod \"autores-model-2ad95a-kserve-58f5587cbf-tgnfr\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:43.614733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.614697 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:33:43.652423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:43.652390 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:44.094232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:44.094200 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr"] Oct 31 15:33:44.094674 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:33:44.094649 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9a0839d_f560_4a73_a294_60e3aa5f94f6.slice/crio-02cc98c975a68a988dbeeb26547e5e4a66865f434adb630f38edbe55622cfbd0 WatchSource:0}: Error finding container 02cc98c975a68a988dbeeb26547e5e4a66865f434adb630f38edbe55622cfbd0: Status 404 returned error can't find the container with id 02cc98c975a68a988dbeeb26547e5e4a66865f434adb630f38edbe55622cfbd0 Oct 31 15:33:44.182557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:44.182513 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" event={"ID":"d9a0839d-f560-4a73-a294-60e3aa5f94f6","Type":"ContainerStarted","Data":"3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa"} Oct 31 15:33:44.182741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:44.182577 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" event={"ID":"d9a0839d-f560-4a73-a294-60e3aa5f94f6","Type":"ContainerStarted","Data":"02cc98c975a68a988dbeeb26547e5e4a66865f434adb630f38edbe55622cfbd0"} Oct 31 15:33:44.653315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:44.653283 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:47.492114 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:33:47.492076 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:33:48.097669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:48.097623 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:52.834592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:52.834554 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:53.653039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:53.653009 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:53.666199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:53.666164 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:53.786973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:53.786915 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" podStartSLOduration=10.786900755 podStartE2EDuration="10.786900755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:33:44.314367856 +0000 UTC m=+2869.933919267" watchObservedRunningTime="2026-10-31 15:33:53.786900755 +0000 UTC m=+2879.406452175" Oct 31 15:33:54.262999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:54.262966 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:33:56.605746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:56.605713 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:57.624573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:57.624538 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:33:58.841583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:33:58.841549 4865 scope.go:122] "RemoveContainer" containerID="72dbf68ee91282e96d772f160937cec4a7854eb2e52f4ef6000b8d5dbca101a1" Oct 31 15:34:01.466048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:01.466011 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:01.955949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:01.955873 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb"] Oct 31 15:34:01.961558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:01.961531 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.041759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.041729 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:02.068438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.068409 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-tmp-dir\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.068621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.068457 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979de514-5322-4324-9a94-46f79c2e6c65-tls-certs\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.068621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.068585 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-model-cache\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.068699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.068653 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57vx\" (UniqueName: \"kubernetes.io/projected/979de514-5322-4324-9a94-46f79c2e6c65-kube-api-access-m57vx\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.068754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.068735 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-home\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.068798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.068789 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-dshm\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.069415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.069398 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf36\"/\"openshift-service-ca.crt\"" Oct 31 15:34:02.069646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.069622 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf36\"/\"kube-root-ca.crt\"" Oct 31 15:34:02.069646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.069640 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf36\"/\"default-dockercfg-sfl4l\"" Oct 31 15:34:02.069805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.069637 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-46bf36\"/\"auth-test-model-46bf36-kserve-self-signed-certs\"" Oct 31 15:34:02.073034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.073017 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb"] Oct 31 15:34:02.170393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170359 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-tmp-dir\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170415 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979de514-5322-4324-9a94-46f79c2e6c65-tls-certs\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170479 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-model-cache\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170577 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57vx\" (UniqueName: \"kubernetes.io/projected/979de514-5322-4324-9a94-46f79c2e6c65-kube-api-access-m57vx\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170623 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-home\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170686 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-dshm\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170830 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-tmp-dir\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170883 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-model-cache\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.170997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.170953 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-home\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.173009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.172985 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-dshm\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.173174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.173156 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979de514-5322-4324-9a94-46f79c2e6c65-tls-certs\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.229999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.229966 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57vx\" (UniqueName: \"kubernetes.io/projected/979de514-5322-4324-9a94-46f79c2e6c65-kube-api-access-m57vx\") pod \"auth-test-model-46bf36-kserve-7f64655fb5-ktklb\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.273727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.273699 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:02.416236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.416201 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:02.659162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.659134 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb"] Oct 31 15:34:02.659661 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:34:02.659637 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod979de514_5322_4324_9a94_46f79c2e6c65.slice/crio-4687bfb81d88a1583fd23831ce5699362ffeaff4787b94751ec4b17509f8fbae WatchSource:0}: Error finding container 4687bfb81d88a1583fd23831ce5699362ffeaff4787b94751ec4b17509f8fbae: Status 404 returned error can't find the container with id 4687bfb81d88a1583fd23831ce5699362ffeaff4787b94751ec4b17509f8fbae Oct 31 15:34:02.830039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:02.829992 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:03.325001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:03.324966 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" event={"ID":"979de514-5322-4324-9a94-46f79c2e6c65","Type":"ContainerStarted","Data":"d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34"} Oct 31 15:34:03.325212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:03.325007 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" event={"ID":"979de514-5322-4324-9a94-46f79c2e6c65","Type":"ContainerStarted","Data":"4687bfb81d88a1583fd23831ce5699362ffeaff4787b94751ec4b17509f8fbae"} Oct 31 15:34:03.444436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:03.444377 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" podStartSLOduration=2.444362901 podStartE2EDuration="2.444362901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:34:03.441163571 +0000 UTC m=+2889.060714982" watchObservedRunningTime="2026-10-31 15:34:03.444362901 +0000 UTC m=+2889.063914311" Oct 31 15:34:04.274731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:04.274694 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:07.210097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:07.210059 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:11.376153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:11.376116 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:12.276493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:12.276020 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:12.309898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:12.309864 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:12.410730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:12.410698 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:34:13.151687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:13.151652 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:15.530427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:15.530387 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:15.672005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:15.671968 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:17.134984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:17.134945 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:17.493243 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:34:17.493203 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:34:17.655789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:17.655756 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx"] Oct 31 15:34:17.656012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:17.655989 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" containerID="cri-o://9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57" gracePeriod=30 Oct 31 15:34:17.687960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:17.687916 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:18.025848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:18.025816 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:18.434442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:18.434346 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:18.484442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:18.484402 4865 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:34:18.484639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:18.484496 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:34:19.009158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:19.009112 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:19.285696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:19.285617 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:22.932085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:22.932019 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:34:23.055864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.055831 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-envoy\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.055876 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1195e70e-0cf4-433c-884d-9c99833cef37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1195e70e-0cf4-433c-884d-9c99833cef37-istio-podinfo\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.055924 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1195e70e-0cf4-433c-884d-9c99833cef37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1195e70e-0cf4-433c-884d-9c99833cef37-istiod-ca-cert\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.055966 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-certs\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056044 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-socket\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056062 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-credential-socket\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056087 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-data\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056230 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-istio-token\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-istio-token\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056267 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-kube-api-access-ld5b8\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-kube-api-access-ld5b8\") pod \"1195e70e-0cf4-433c-884d-9c99833cef37\" (UID: \"1195e70e-0cf4-433c-884d-9c99833cef37\") " Oct 31 15:34:23.056431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056307 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-certs" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.056431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056389 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-socket" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.056937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056542 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-data" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.056937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056706 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-credential-socket" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.056937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.056863 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1195e70e-0cf4-433c-884d-9c99833cef37-istiod-ca-cert" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:34:23.057437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.057145 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1195e70e-0cf4-433c-884d-9c99833cef37-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.057437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.057167 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.057437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.057179 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.057437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.057191 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.057437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.057203 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.058933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.058866 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-envoy" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.059441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.059403 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-istio-token" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:34:23.059770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.059741 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-kube-api-access-ld5b8" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "kube-api-access-ld5b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:34:23.060443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.060417 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1195e70e-0cf4-433c-884d-9c99833cef37-istio-podinfo" pod "1195e70e-0cf4-433c-884d-9c99833cef37" (UID: "1195e70e-0cf4-433c-884d-9c99833cef37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:34:23.158560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.158530 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.158560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.158557 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5b8\" (UniqueName: \"kubernetes.io/projected/1195e70e-0cf4-433c-884d-9c99833cef37-kube-api-access-ld5b8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.158786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.158569 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1195e70e-0cf4-433c-884d-9c99833cef37-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.158786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.158577 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1195e70e-0cf4-433c-884d-9c99833cef37-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.236981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.236944 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:23.477631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.477594 4865 generic.go:356] "Generic (PLEG): container finished" podID="1195e70e-0cf4-433c-884d-9c99833cef37" containerID="9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57" exitCode=0 Oct 31 15:34:23.477859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.477643 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" Oct 31 15:34:23.477859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.477649 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" event={"ID":"1195e70e-0cf4-433c-884d-9c99833cef37","Type":"ContainerDied","Data":"9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57"} Oct 31 15:34:23.477859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.477705 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx" event={"ID":"1195e70e-0cf4-433c-884d-9c99833cef37","Type":"ContainerDied","Data":"c7ba3b4c14b0e8dca63a38958223a0d2590ea60f6a7b9e6985261e84faa05c90"} Oct 31 15:34:23.477859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.477734 4865 scope.go:122] "RemoveContainer" containerID="9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57" Oct 31 15:34:23.492701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.492676 4865 scope.go:122] "RemoveContainer" containerID="66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf" Oct 31 15:34:23.503024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.503002 4865 scope.go:122] "RemoveContainer" containerID="9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57" Oct 31 15:34:23.503304 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:34:23.503283 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57\": container with ID starting with 9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57 not found: ID does not exist" containerID="9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57" Oct 31 15:34:23.503376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.503312 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57"} err="failed to get container status \"9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57\": rpc error: code = NotFound desc = could not find container \"9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57\": container with ID starting with 9db180672db732cb9e68f9e2d8d379777321305a70c9bd38586c55658ca1ac57 not found: ID does not exist" Oct 31 15:34:23.503376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.503338 4865 scope.go:122] "RemoveContainer" containerID="66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf" Oct 31 15:34:23.503712 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:34:23.503689 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf\": container with ID starting with 66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf not found: ID does not exist" containerID="66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf" Oct 31 15:34:23.503776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.503718 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf"} err="failed to get container status \"66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf\": rpc error: code = NotFound desc = could not find container \"66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf\": container with ID starting with 66beb7428c4b0d68fbaf57f2bc734c71271eb44f91c99f2a67da62b70b268baf not found: ID does not exist" Oct 31 15:34:23.649818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.649788 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx"] Oct 31 15:34:23.652742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.652718 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-dx7tx"] Oct 31 15:34:23.913870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:23.913757 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:24.098042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:24.098010 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:24.240685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:24.240644 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:24.506108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:24.506023 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" path="/var/lib/kubelet/pods/1195e70e-0cf4-433c-884d-9c99833cef37/volumes" Oct 31 15:34:26.424851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:26.424817 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:34:28.119085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.119039 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:28.299933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.299238 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:28.454917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.454807 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:28.531894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.531864 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl"] Oct 31 15:34:28.532449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.532436 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" Oct 31 15:34:28.532527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.532450 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" Oct 31 15:34:28.532610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.532599 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" Oct 31 15:34:28.532647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.532615 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" Oct 31 15:34:28.537323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.537304 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.593113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.593085 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl"] Oct 31 15:34:28.652494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.651756 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-24ca6e\"" Oct 31 15:34:28.652494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.651963 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-24ca6e-dockercfg-nwcx4\"" Oct 31 15:34:28.673228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.673192 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900648a-b48a-4869-a233-017d634d6e9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.673447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.673281 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/7900648a-b48a-4869-a233-017d634d6e9d-kube-api-access-75qt7\") pod \"payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.720546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.707311 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4"] Oct 31 15:34:28.720546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.708345 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" Oct 31 15:34:28.720546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.708369 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1195e70e-0cf4-433c-884d-9c99833cef37" containerName="istio-proxy" Oct 31 15:34:28.720994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.720746 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:28.768551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.767353 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4"] Oct 31 15:34:28.774769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.774740 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900648a-b48a-4869-a233-017d634d6e9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.774920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.774796 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/7900648a-b48a-4869-a233-017d634d6e9d-kube-api-access-75qt7\") pod \"payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.775512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.775493 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900648a-b48a-4869-a233-017d634d6e9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.833282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.833243 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/7900648a-b48a-4869-a233-017d634d6e9d-kube-api-access-75qt7\") pod \"payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.852143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.852113 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:28.875970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.875939 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqvb\" (UniqueName: \"kubernetes.io/projected/aa9aa055-5783-4f2f-b696-7531f88f4f6e-kube-api-access-rdqvb\") pod \"payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:28.876106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.875987 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9aa055-5783-4f2f-b696-7531f88f4f6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:28.977438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.977351 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqvb\" (UniqueName: \"kubernetes.io/projected/aa9aa055-5783-4f2f-b696-7531f88f4f6e-kube-api-access-rdqvb\") pod \"payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:28.977438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.977405 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9aa055-5783-4f2f-b696-7531f88f4f6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:28.978417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:28.978392 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9aa055-5783-4f2f-b696-7531f88f4f6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:29.037166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.037127 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqvb\" (UniqueName: \"kubernetes.io/projected/aa9aa055-5783-4f2f-b696-7531f88f4f6e-kube-api-access-rdqvb\") pod \"payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:29.051753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.051724 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:29.230295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.230267 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl"] Oct 31 15:34:29.230717 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:34:29.230644 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7900648a_b48a_4869_a233_017d634d6e9d.slice/crio-df36810c42f17310df5a846d5e62260173364a1308c5c919336ca0cbe9d91462 WatchSource:0}: Error finding container df36810c42f17310df5a846d5e62260173364a1308c5c919336ca0cbe9d91462: Status 404 returned error can't find the container with id df36810c42f17310df5a846d5e62260173364a1308c5c919336ca0cbe9d91462 Oct 31 15:34:29.427419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.427389 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4"] Oct 31 15:34:29.427680 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:34:29.427654 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9aa055_5783_4f2f_b696_7531f88f4f6e.slice/crio-e982e8f97a5124a5b609a765f3cace7698dedf9a4ebfee7f54d1620c1cda0275 WatchSource:0}: Error finding container e982e8f97a5124a5b609a765f3cace7698dedf9a4ebfee7f54d1620c1cda0275: Status 404 returned error can't find the container with id e982e8f97a5124a5b609a765f3cace7698dedf9a4ebfee7f54d1620c1cda0275 Oct 31 15:34:29.522633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.522592 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" event={"ID":"aa9aa055-5783-4f2f-b696-7531f88f4f6e","Type":"ContainerStarted","Data":"3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e"} Oct 31 15:34:29.522840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.522658 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" event={"ID":"aa9aa055-5783-4f2f-b696-7531f88f4f6e","Type":"ContainerStarted","Data":"e982e8f97a5124a5b609a765f3cace7698dedf9a4ebfee7f54d1620c1cda0275"} Oct 31 15:34:29.522840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.522683 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:29.524509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.524480 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" event={"ID":"7900648a-b48a-4869-a233-017d634d6e9d","Type":"ContainerStarted","Data":"7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6"} Oct 31 15:34:29.524644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.524525 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" event={"ID":"7900648a-b48a-4869-a233-017d634d6e9d","Type":"ContainerStarted","Data":"df36810c42f17310df5a846d5e62260173364a1308c5c919336ca0cbe9d91462"} Oct 31 15:34:29.524644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.524590 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:29.639225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.639172 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" podStartSLOduration=1.639155103 podStartE2EDuration="1.639155103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:34:29.638503219 +0000 UTC m=+2915.258054631" watchObservedRunningTime="2026-10-31 15:34:29.639155103 +0000 UTC m=+2915.258706515" Oct 31 15:34:29.770668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:29.770545 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" podStartSLOduration=1.7705255960000001 podStartE2EDuration="1.770525596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:34:29.76637305 +0000 UTC m=+2915.385924463" watchObservedRunningTime="2026-10-31 15:34:29.770525596 +0000 UTC m=+2915.390077073" Oct 31 15:34:33.221389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:33.221355 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:37.387780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:37.387691 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr"] Oct 31 15:34:37.400184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:37.400152 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:37.803732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:37.803698 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:38.143813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:38.143723 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:40.534751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:40.534723 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:34:40.535167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:40.534797 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:34:42.863023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:42.862990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:42.991249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:42.991214 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:47.490823 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:34:47.490789 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:34:47.595511 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:47.595412 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:52.413287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.413246 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" podUID="d9a0839d-f560-4a73-a294-60e3aa5f94f6" containerName="main" containerID="cri-o://3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa" gracePeriod=45 Oct 31 15:34:52.428411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.428384 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:52.693537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.693512 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:34:52.700586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.700550 4865 generic.go:356] "Generic (PLEG): container finished" podID="d9a0839d-f560-4a73-a294-60e3aa5f94f6" containerID="3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa" exitCode=0 Oct 31 15:34:52.700732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.700597 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" Oct 31 15:34:52.700732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.700630 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" event={"ID":"d9a0839d-f560-4a73-a294-60e3aa5f94f6","Type":"ContainerDied","Data":"3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa"} Oct 31 15:34:52.700732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.700673 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr" event={"ID":"d9a0839d-f560-4a73-a294-60e3aa5f94f6","Type":"ContainerDied","Data":"02cc98c975a68a988dbeeb26547e5e4a66865f434adb630f38edbe55622cfbd0"} Oct 31 15:34:52.700732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.700695 4865 scope.go:122] "RemoveContainer" containerID="3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa" Oct 31 15:34:52.710720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.710696 4865 scope.go:122] "RemoveContainer" containerID="3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa" Oct 31 15:34:52.710979 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:34:52.710952 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa\": container with ID starting with 3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa not found: ID does not exist" containerID="3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa" Oct 31 15:34:52.711054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.710991 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa"} err="failed to get container status \"3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa\": rpc error: code = NotFound desc = could not find container \"3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa\": container with ID starting with 3361f2670625f74d9df7177a8dd3c8181343fb9869df363375d9e5c8e0c035fa not found: ID does not exist" Oct 31 15:34:52.783257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783231 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-model-cache\") pod \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " Oct 31 15:34:52.783389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783297 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-dshm\") pod \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " Oct 31 15:34:52.783389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783349 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a0839d-f560-4a73-a294-60e3aa5f94f6-kube-api-access-gfbb5\" (UniqueName: \"kubernetes.io/projected/d9a0839d-f560-4a73-a294-60e3aa5f94f6-kube-api-access-gfbb5\") pod \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " Oct 31 15:34:52.783548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783404 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tmp-dir\") pod \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " Oct 31 15:34:52.783548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783443 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-home\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-home\") pod \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " Oct 31 15:34:52.783548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783501 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-model-cache" pod "d9a0839d-f560-4a73-a294-60e3aa5f94f6" (UID: "d9a0839d-f560-4a73-a294-60e3aa5f94f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:52.783731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783589 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tls-certs\") pod \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\" (UID: \"d9a0839d-f560-4a73-a294-60e3aa5f94f6\") " Oct 31 15:34:52.783794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783754 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tmp-dir" pod "d9a0839d-f560-4a73-a294-60e3aa5f94f6" (UID: "d9a0839d-f560-4a73-a294-60e3aa5f94f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:52.783794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.783768 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-home" pod "d9a0839d-f560-4a73-a294-60e3aa5f94f6" (UID: "d9a0839d-f560-4a73-a294-60e3aa5f94f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:52.784296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.784274 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:52.784296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.784297 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:52.784457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.784306 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:52.785710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.785690 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-dshm" pod "d9a0839d-f560-4a73-a294-60e3aa5f94f6" (UID: "d9a0839d-f560-4a73-a294-60e3aa5f94f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:52.785821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.785794 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tls-certs" pod "d9a0839d-f560-4a73-a294-60e3aa5f94f6" (UID: "d9a0839d-f560-4a73-a294-60e3aa5f94f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:34:52.785950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.785937 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a0839d-f560-4a73-a294-60e3aa5f94f6-kube-api-access-gfbb5" pod "d9a0839d-f560-4a73-a294-60e3aa5f94f6" (UID: "d9a0839d-f560-4a73-a294-60e3aa5f94f6"). InnerVolumeSpecName "kube-api-access-gfbb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:34:52.885349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.885320 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfbb5\" (UniqueName: \"kubernetes.io/projected/d9a0839d-f560-4a73-a294-60e3aa5f94f6-kube-api-access-gfbb5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:52.885349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.885348 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a0839d-f560-4a73-a294-60e3aa5f94f6-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:52.885555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:52.885358 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9a0839d-f560-4a73-a294-60e3aa5f94f6-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:53.126648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:53.126615 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr"] Oct 31 15:34:53.132075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:53.132053 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2ad95a/autores-model-2ad95a-kserve-58f5587cbf-tgnfr"] Oct 31 15:34:54.497538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:54.497510 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a0839d-f560-4a73-a294-60e3aa5f94f6" path="/var/lib/kubelet/pods/d9a0839d-f560-4a73-a294-60e3aa5f94f6/volumes" Oct 31 15:34:57.133199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:57.133160 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:34:58.884224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:34:58.884192 4865 scope.go:122] "RemoveContainer" containerID="95d58c967483aa6d6151507b0098223d782a093a3dbb7291f362e035b877c720" Oct 31 15:35:01.028158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:01.028120 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:35:01.733597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:01.733559 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:01.895063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:01.895018 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:03.172829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:03.172788 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:06.550412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:06.550339 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:11.275249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:11.275202 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:13.941763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:13.941726 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595"] Oct 31 15:35:13.942185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:13.941971 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" containerID="cri-o://e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15" gracePeriod=30 Oct 31 15:35:16.000053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:16.000012 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:17.491264 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:35:17.491227 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:35:17.908391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:17.908308 4865 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:35:17.908391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:17.908366 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:35:19.218065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.217995 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:35:19.333019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.332990 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-token\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-token\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333037 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-data\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333163 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-envoy\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333224 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-credential-socket\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333309 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-certs\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333365 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-socket\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333361 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-data" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:19.333425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333409 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-credential-socket" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:19.333693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333460 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-podinfo\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333546 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-kube-api-access-88b46\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-kube-api-access-88b46\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.333693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333582 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-certs" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:19.333693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333599 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-socket" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:19.333693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333603 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a6ff32-2bcb-4688-9844-48856a844dc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a6ff32-2bcb-4688-9844-48856a844dc7-istiod-ca-cert\") pod \"a7a6ff32-2bcb-4688-9844-48856a844dc7\" (UID: \"a7a6ff32-2bcb-4688-9844-48856a844dc7\") " Oct 31 15:35:19.334146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.333932 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a6ff32-2bcb-4688-9844-48856a844dc7-istiod-ca-cert" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:35:19.334255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.334230 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a6ff32-2bcb-4688-9844-48856a844dc7-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.334320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.334257 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.334320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.334273 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.334320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.334287 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.334320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.334300 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.335763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.335733 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-envoy" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:19.335919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.335902 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-kube-api-access-88b46" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "kube-api-access-88b46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:35:19.336096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.336077 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-podinfo" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:35:19.336202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.336184 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-token" pod "a7a6ff32-2bcb-4688-9844-48856a844dc7" (UID: "a7a6ff32-2bcb-4688-9844-48856a844dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:35:19.435751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.435713 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.435751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.435746 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.435751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.435758 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88b46\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-kube-api-access-88b46\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.436020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.435767 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a6ff32-2bcb-4688-9844-48856a844dc7-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:19.919355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.919317 4865 generic.go:356] "Generic (PLEG): container finished" podID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerID="e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15" exitCode=0 Oct 31 15:35:19.919605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.919375 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" Oct 31 15:35:19.919605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.919407 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" event={"ID":"a7a6ff32-2bcb-4688-9844-48856a844dc7","Type":"ContainerDied","Data":"e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15"} Oct 31 15:35:19.919605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.919492 4865 scope.go:122] "RemoveContainer" containerID="e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15" Oct 31 15:35:19.919605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.919503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595" event={"ID":"a7a6ff32-2bcb-4688-9844-48856a844dc7","Type":"ContainerDied","Data":"26afad7a1fd5b4c21dad5d05632428f325bdc251984dcbbf437b2c9b2f048bdd"} Oct 31 15:35:19.930605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.930588 4865 scope.go:122] "RemoveContainer" containerID="e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15" Oct 31 15:35:19.930898 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:35:19.930880 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15\": container with ID starting with e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15 not found: ID does not exist" containerID="e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15" Oct 31 15:35:19.930959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:19.930904 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15"} err="failed to get container status \"e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15\": rpc error: code = NotFound desc = could not find container \"e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15\": container with ID starting with e0320db7f2b07baee05269542098dc4a460824d379864f44cb49f6761b984b15 not found: ID does not exist" Oct 31 15:35:20.140392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.140353 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595"] Oct 31 15:35:20.142868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.142834 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:20.191576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.191463 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-kd595"] Oct 31 15:35:20.456835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.456757 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n"] Oct 31 15:35:20.498006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.497973 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" path="/var/lib/kubelet/pods/a7a6ff32-2bcb-4688-9844-48856a844dc7/volumes" Oct 31 15:35:20.566042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.565977 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8"] Oct 31 15:35:20.814227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.814189 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:20.975344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:20.975305 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:25.458074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.458031 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" podUID="80207be4-6a62-423c-9691-a64155ace7a9" containerName="payload-processing" containerID="cri-o://ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97" gracePeriod=25 Oct 31 15:35:25.566752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.566711 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" podUID="b528266d-5fc3-4ccb-9e7a-a53fe13f15de" containerName="payload-pre-processing" containerID="cri-o://625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a" gracePeriod=25 Oct 31 15:35:25.653362 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.653327 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:25.762116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.762088 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:35:25.861741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.861708 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:25.861859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.861780 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:35:25.921853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.921813 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80207be4-6a62-423c-9691-a64155ace7a9-kube-api-access-klnh8\" (UniqueName: \"kubernetes.io/projected/80207be4-6a62-423c-9691-a64155ace7a9-kube-api-access-klnh8\") pod \"80207be4-6a62-423c-9691-a64155ace7a9\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " Oct 31 15:35:25.922048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.921940 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-kube-api-access-cghhf\" (UniqueName: \"kubernetes.io/projected/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-kube-api-access-cghhf\") pod \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " Oct 31 15:35:25.922435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.922389 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80207be4-6a62-423c-9691-a64155ace7a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80207be4-6a62-423c-9691-a64155ace7a9-plugins-config-volume\") pod \"80207be4-6a62-423c-9691-a64155ace7a9\" (UID: \"80207be4-6a62-423c-9691-a64155ace7a9\") " Oct 31 15:35:25.922758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.922453 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-plugins-config-volume\") pod \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\" (UID: \"b528266d-5fc3-4ccb-9e7a-a53fe13f15de\") " Oct 31 15:35:25.922758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.922648 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80207be4-6a62-423c-9691-a64155ace7a9-plugins-config-volume" pod "80207be4-6a62-423c-9691-a64155ace7a9" (UID: "80207be4-6a62-423c-9691-a64155ace7a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:35:25.922965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.922935 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-plugins-config-volume" pod "b528266d-5fc3-4ccb-9e7a-a53fe13f15de" (UID: "b528266d-5fc3-4ccb-9e7a-a53fe13f15de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:35:25.923163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.923141 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80207be4-6a62-423c-9691-a64155ace7a9-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:25.923243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.923170 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:25.924080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.924052 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80207be4-6a62-423c-9691-a64155ace7a9-kube-api-access-klnh8" pod "80207be4-6a62-423c-9691-a64155ace7a9" (UID: "80207be4-6a62-423c-9691-a64155ace7a9"). InnerVolumeSpecName "kube-api-access-klnh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:35:25.924287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.924265 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-kube-api-access-cghhf" pod "b528266d-5fc3-4ccb-9e7a-a53fe13f15de" (UID: "b528266d-5fc3-4ccb-9e7a-a53fe13f15de"). InnerVolumeSpecName "kube-api-access-cghhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:35:25.962399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.962373 4865 generic.go:356] "Generic (PLEG): container finished" podID="80207be4-6a62-423c-9691-a64155ace7a9" containerID="ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97" exitCode=0 Oct 31 15:35:25.962580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.962427 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" Oct 31 15:35:25.962580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.962456 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" event={"ID":"80207be4-6a62-423c-9691-a64155ace7a9","Type":"ContainerDied","Data":"ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97"} Oct 31 15:35:25.962580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.962541 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n" event={"ID":"80207be4-6a62-423c-9691-a64155ace7a9","Type":"ContainerDied","Data":"c1e3dc5a529639c953f2bc2243b015ef46a50960279de58dc6ce057f9811dcf8"} Oct 31 15:35:25.962580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.962571 4865 scope.go:122] "RemoveContainer" containerID="ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97" Oct 31 15:35:25.963873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.963851 4865 generic.go:356] "Generic (PLEG): container finished" podID="b528266d-5fc3-4ccb-9e7a-a53fe13f15de" containerID="625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a" exitCode=0 Oct 31 15:35:25.963959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.963907 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" Oct 31 15:35:25.963959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.963930 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" event={"ID":"b528266d-5fc3-4ccb-9e7a-a53fe13f15de","Type":"ContainerDied","Data":"625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a"} Oct 31 15:35:25.964066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.963982 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8" event={"ID":"b528266d-5fc3-4ccb-9e7a-a53fe13f15de","Type":"ContainerDied","Data":"07d0f4757897243248df99212174c54ad8f1b2f1ef3f8b254f532490a55c309d"} Oct 31 15:35:25.972888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.972868 4865 scope.go:122] "RemoveContainer" containerID="ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97" Oct 31 15:35:25.973195 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:35:25.973175 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97\": container with ID starting with ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97 not found: ID does not exist" containerID="ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97" Oct 31 15:35:25.973281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.973202 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97"} err="failed to get container status \"ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97\": rpc error: code = NotFound desc = could not find container \"ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97\": container with ID starting with ff7c3c6177c374cd0f894144ea6777343e4bb60f39a1b336e4d3a9c444691e97 not found: ID does not exist" Oct 31 15:35:25.973281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.973217 4865 scope.go:122] "RemoveContainer" containerID="625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a" Oct 31 15:35:25.982610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.982581 4865 scope.go:122] "RemoveContainer" containerID="625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a" Oct 31 15:35:25.982844 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:35:25.982826 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a\": container with ID starting with 625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a not found: ID does not exist" containerID="625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a" Oct 31 15:35:25.982894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:25.982859 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a"} err="failed to get container status \"625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a\": rpc error: code = NotFound desc = could not find container \"625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a\": container with ID starting with 625e28f0559b9207f3eec60c5759535cec215539b613d634b817f2990899ea6a not found: ID does not exist" Oct 31 15:35:26.002443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.002395 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:26.024458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.024429 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnh8\" (UniqueName: \"kubernetes.io/projected/80207be4-6a62-423c-9691-a64155ace7a9-kube-api-access-klnh8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:26.024583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.024463 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghhf\" (UniqueName: \"kubernetes.io/projected/b528266d-5fc3-4ccb-9e7a-a53fe13f15de-kube-api-access-cghhf\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:26.115420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.115385 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n"] Oct 31 15:35:26.118718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.118697 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2ad95a-7d7b8c5bb5-c967n"] Oct 31 15:35:26.228894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.228864 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8"] Oct 31 15:35:26.234070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.234013 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2ad95a-59cd457ff5-xtbt8"] Oct 31 15:35:26.247506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.247461 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:26.498006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.497940 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80207be4-6a62-423c-9691-a64155ace7a9" path="/var/lib/kubelet/pods/80207be4-6a62-423c-9691-a64155ace7a9/volumes" Oct 31 15:35:26.498397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:26.498383 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b528266d-5fc3-4ccb-9e7a-a53fe13f15de" path="/var/lib/kubelet/pods/b528266d-5fc3-4ccb-9e7a-a53fe13f15de/volumes" Oct 31 15:35:30.835590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:30.835556 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:35.608226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:35.608193 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:40.330293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:40.330225 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:43.571201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:43.571165 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:35:45.026347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:45.026310 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:47.491314 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:35:47.491279 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:35:49.301022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:49.300986 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:51.160493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:51.160446 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:51.801247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:51.801215 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:55.984347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:55.984321 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:35:55.992759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:55.992739 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:35:56.053014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:56.052986 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:57.585360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:57.585321 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:57.725939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:57.725904 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:58.412427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:58.412394 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:35:58.421278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:58.421257 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:35:59.245973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:59.245937 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:35:59.853177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:59.853145 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt"] Oct 31 15:35:59.853404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:59.853380 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" containerID="cri-o://7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee" gracePeriod=30 Oct 31 15:35:59.869111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:59.869088 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9"] Oct 31 15:35:59.869297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:59.869277 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" containerID="cri-o://a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8" gracePeriod=30 Oct 31 15:35:59.987072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:35:59.987028 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:00.492407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:00.492371 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:01.113083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:01.113048 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:01.386579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:01.386396 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:01.637198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:01.637111 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:02.066607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:02.066568 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:03.098069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:03.098039 4865 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:36:03.098489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:03.098093 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:36:03.510989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:03.510931 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:05.136301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.136236 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:36:05.166027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.166006 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:36:05.205212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205183 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-kube-api-access-dgzgd\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-kube-api-access-dgzgd\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.205347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205252 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-socket\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.205347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205299 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-envoy\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.205517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205360 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-kube-api-access-6xvgl\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-kube-api-access-6xvgl\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.205517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205417 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-data\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.205517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205462 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785a65e1-79df-46fd-94dc-f8118b3b2f73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785a65e1-79df-46fd-94dc-f8118b3b2f73-istiod-ca-cert\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.205694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205534 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-token\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-token\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.205694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205577 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-credential-socket\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.205694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205643 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-podinfo\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.205868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205700 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-credential-socket\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.205868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205744 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-podinfo\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.205868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205779 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-socket\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.206038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205582 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-socket" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.206038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205755 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-data" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.206038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205883 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-credential-socket" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.206038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205906 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-certs\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.206038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.205985 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-data\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.206332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206128 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785a65e1-79df-46fd-94dc-f8118b3b2f73-istiod-ca-cert" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:36:05.206332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206142 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-certs" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.206332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206177 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-certs\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.206332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206308 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-envoy\") pod \"785a65e1-79df-46fd-94dc-f8118b3b2f73\" (UID: \"785a65e1-79df-46fd-94dc-f8118b3b2f73\") " Oct 31 15:36:05.206607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206411 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-token\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-token\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.206607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206489 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ea74b0-0d65-441e-8f8e-1274754236d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ea74b0-0d65-441e-8f8e-1274754236d3-istiod-ca-cert\") pod \"d3ea74b0-0d65-441e-8f8e-1274754236d3\" (UID: \"d3ea74b0-0d65-441e-8f8e-1274754236d3\") " Oct 31 15:36:05.206717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206681 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-credential-socket" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.207146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.206899 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-data" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.207146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207108 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-certs" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.207146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207037 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ea74b0-0d65-441e-8f8e-1274754236d3-istiod-ca-cert" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207190 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-socket" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207332 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785a65e1-79df-46fd-94dc-f8118b3b2f73-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207354 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207369 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207383 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207398 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207413 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207429 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207444 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ea74b0-0d65-441e-8f8e-1274754236d3-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207458 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.207641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.207496 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.208922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.208900 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-token" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.209018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.208952 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-kube-api-access-dgzgd" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "kube-api-access-dgzgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.209018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.208974 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-kube-api-access-6xvgl" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "kube-api-access-6xvgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.209143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.209093 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-podinfo" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:36:05.209198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.209138 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-token" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.209360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.209336 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-podinfo" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:36:05.209424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.209357 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-envoy" pod "d3ea74b0-0d65-441e-8f8e-1274754236d3" (UID: "d3ea74b0-0d65-441e-8f8e-1274754236d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.209779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.209764 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-envoy" pod "785a65e1-79df-46fd-94dc-f8118b3b2f73" (UID: "785a65e1-79df-46fd-94dc-f8118b3b2f73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.229554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.229495 4865 generic.go:356] "Generic (PLEG): container finished" podID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerID="7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee" exitCode=0 Oct 31 15:36:05.229554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.229535 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" Oct 31 15:36:05.229707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.229590 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" event={"ID":"d3ea74b0-0d65-441e-8f8e-1274754236d3","Type":"ContainerDied","Data":"7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee"} Oct 31 15:36:05.229707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.229650 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt" event={"ID":"d3ea74b0-0d65-441e-8f8e-1274754236d3","Type":"ContainerDied","Data":"f43d1bafc276dc99f7faea0a0dfb24a4c1e99730ac1d5d904753194883be1aa2"} Oct 31 15:36:05.229707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.229679 4865 scope.go:122] "RemoveContainer" containerID="7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee" Oct 31 15:36:05.231545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.231523 4865 generic.go:356] "Generic (PLEG): container finished" podID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerID="a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8" exitCode=0 Oct 31 15:36:05.231663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.231586 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" event={"ID":"785a65e1-79df-46fd-94dc-f8118b3b2f73","Type":"ContainerDied","Data":"a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8"} Oct 31 15:36:05.231663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.231616 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" event={"ID":"785a65e1-79df-46fd-94dc-f8118b3b2f73","Type":"ContainerDied","Data":"9b89789223b13447d0373155c6a3d9e88f58718276de59191b79eb85f1934640"} Oct 31 15:36:05.231663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.231585 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9" Oct 31 15:36:05.240041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.240021 4865 scope.go:122] "RemoveContainer" containerID="7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee" Oct 31 15:36:05.240388 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:36:05.240296 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee\": container with ID starting with 7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee not found: ID does not exist" containerID="7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee" Oct 31 15:36:05.240388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.240327 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee"} err="failed to get container status \"7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee\": rpc error: code = NotFound desc = could not find container \"7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee\": container with ID starting with 7a191778662d1a86106fb454d3597b400abb64da5f75f6312b89d16afdab99ee not found: ID does not exist" Oct 31 15:36:05.240388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.240353 4865 scope.go:122] "RemoveContainer" containerID="a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8" Oct 31 15:36:05.249864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.249844 4865 scope.go:122] "RemoveContainer" containerID="4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a" Oct 31 15:36:05.258458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.258444 4865 scope.go:122] "RemoveContainer" containerID="a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8" Oct 31 15:36:05.258761 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:36:05.258738 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8\": container with ID starting with a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8 not found: ID does not exist" containerID="a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8" Oct 31 15:36:05.258807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.258770 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8"} err="failed to get container status \"a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8\": rpc error: code = NotFound desc = could not find container \"a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8\": container with ID starting with a80680f44aa69bbd05c23e378a78ef0720944521434237dd2c29096f2b109df8 not found: ID does not exist" Oct 31 15:36:05.258807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.258789 4865 scope.go:122] "RemoveContainer" containerID="4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a" Oct 31 15:36:05.259054 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:36:05.259037 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a\": container with ID starting with 4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a not found: ID does not exist" containerID="4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a" Oct 31 15:36:05.259094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.259070 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a"} err="failed to get container status \"4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a\": rpc error: code = NotFound desc = could not find container \"4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a\": container with ID starting with 4b8e9f375a31d7ae3279acd26d2fecfb6f592aee2f667eca7326b3dab2c1af6a not found: ID does not exist" Oct 31 15:36:05.308854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308829 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.308854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308855 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.309010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308866 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzgd\" (UniqueName: \"kubernetes.io/projected/d3ea74b0-0d65-441e-8f8e-1274754236d3-kube-api-access-dgzgd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.309010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308874 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.309010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308883 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xvgl\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-kube-api-access-6xvgl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.309010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308892 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.309010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308903 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785a65e1-79df-46fd-94dc-f8118b3b2f73-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.309010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.308916 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ea74b0-0d65-441e-8f8e-1274754236d3-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.492067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.492033 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9"] Oct 31 15:36:05.497047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.497015 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-p6fb9"] Oct 31 15:36:05.608787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.608755 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt"] Oct 31 15:36:05.612931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.612908 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-wrrjt"] Oct 31 15:36:05.666726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.666692 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:05.855046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.854964 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:05.995974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:05.995943 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:06.226575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:06.226535 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:06.502957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:06.501950 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" path="/var/lib/kubelet/pods/785a65e1-79df-46fd-94dc-f8118b3b2f73/volumes" Oct 31 15:36:06.502957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:06.502786 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" path="/var/lib/kubelet/pods/d3ea74b0-0d65-441e-8f8e-1274754236d3/volumes" Oct 31 15:36:08.870247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:08.870216 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:09.038609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.038575 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:09.288217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288183 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c"] Oct 31 15:36:09.288804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288788 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b528266d-5fc3-4ccb-9e7a-a53fe13f15de" containerName="payload-pre-processing" Oct 31 15:36:09.288871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288806 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b528266d-5fc3-4ccb-9e7a-a53fe13f15de" containerName="payload-pre-processing" Oct 31 15:36:09.288871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288825 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" Oct 31 15:36:09.288871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288831 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" Oct 31 15:36:09.288871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288844 4865 cpu_manager.go:414] "containerMap: removing container" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" Oct 31 15:36:09.288871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288851 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" Oct 31 15:36:09.288871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288867 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" Oct 31 15:36:09.288871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288872 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288879 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d9a0839d-f560-4a73-a294-60e3aa5f94f6" containerName="main" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288884 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a0839d-f560-4a73-a294-60e3aa5f94f6" containerName="main" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288900 4865 cpu_manager.go:414] "containerMap: removing container" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288905 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288914 4865 cpu_manager.go:414] "containerMap: removing container" podUID="80207be4-6a62-423c-9691-a64155ace7a9" containerName="payload-processing" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.288919 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80207be4-6a62-423c-9691-a64155ace7a9" containerName="payload-processing" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.289008 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.289015 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="80207be4-6a62-423c-9691-a64155ace7a9" containerName="payload-processing" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.289026 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b528266d-5fc3-4ccb-9e7a-a53fe13f15de" containerName="payload-pre-processing" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.289033 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a6ff32-2bcb-4688-9844-48856a844dc7" containerName="istio-proxy" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.289039 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="785a65e1-79df-46fd-94dc-f8118b3b2f73" containerName="istio-proxy" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.289045 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ea74b0-0d65-441e-8f8e-1274754236d3" containerName="istio-proxy" Oct 31 15:36:09.289110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.289051 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a0839d-f560-4a73-a294-60e3aa5f94f6" containerName="main" Oct 31 15:36:09.293861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.293840 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.356128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.356096 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c"] Oct 31 15:36:09.364049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.361225 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed24e4-2bc0-4510-acba-e13bcac34b49-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.364049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.361395 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2j5\" (UniqueName: \"kubernetes.io/projected/38ed24e4-2bc0-4510-acba-e13bcac34b49-kube-api-access-xt2j5\") pod \"payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.403496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.402576 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2cb1b7-dockercfg-67mfd\"" Oct 31 15:36:09.403496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.402670 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2cb1b7\"" Oct 31 15:36:09.463136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.463095 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed24e4-2bc0-4510-acba-e13bcac34b49-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.463296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.463285 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2j5\" (UniqueName: \"kubernetes.io/projected/38ed24e4-2bc0-4510-acba-e13bcac34b49-kube-api-access-xt2j5\") pod \"payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.464018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.463983 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed24e4-2bc0-4510-acba-e13bcac34b49-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.465965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.465941 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66"] Oct 31 15:36:09.470193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.470176 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.529912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.529881 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2j5\" (UniqueName: \"kubernetes.io/projected/38ed24e4-2bc0-4510-acba-e13bcac34b49-kube-api-access-xt2j5\") pod \"payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.534285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.534247 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66"] Oct 31 15:36:09.564785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.564708 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwm2z\" (UniqueName: \"kubernetes.io/projected/865c079d-7de8-4e62-98b6-4f06f5cef32e-kube-api-access-nwm2z\") pod \"payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.564950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.564826 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865c079d-7de8-4e62-98b6-4f06f5cef32e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.610234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.610211 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:09.668641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.668501 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwm2z\" (UniqueName: \"kubernetes.io/projected/865c079d-7de8-4e62-98b6-4f06f5cef32e-kube-api-access-nwm2z\") pod \"payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.668641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.668622 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865c079d-7de8-4e62-98b6-4f06f5cef32e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.669735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.669704 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865c079d-7de8-4e62-98b6-4f06f5cef32e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.728578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.728542 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwm2z\" (UniqueName: \"kubernetes.io/projected/865c079d-7de8-4e62-98b6-4f06f5cef32e-kube-api-access-nwm2z\") pod \"payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.783955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.783923 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:09.988624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:09.988591 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c"] Oct 31 15:36:09.989001 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:36:09.988868 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ed24e4_2bc0_4510_acba_e13bcac34b49.slice/crio-4ef99425ceab0eb96d510ef1e8fcf94442965d85faf0d9705189ec6f2e8da2c7 WatchSource:0}: Error finding container 4ef99425ceab0eb96d510ef1e8fcf94442965d85faf0d9705189ec6f2e8da2c7: Status 404 returned error can't find the container with id 4ef99425ceab0eb96d510ef1e8fcf94442965d85faf0d9705189ec6f2e8da2c7 Oct 31 15:36:10.164334 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:36:10.164307 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865c079d_7de8_4e62_98b6_4f06f5cef32e.slice/crio-1e4912e84f35377f33f2d8aeed1518b05924ad1fd3a17a366d75123ec4e01fe7 WatchSource:0}: Error finding container 1e4912e84f35377f33f2d8aeed1518b05924ad1fd3a17a366d75123ec4e01fe7: Status 404 returned error can't find the container with id 1e4912e84f35377f33f2d8aeed1518b05924ad1fd3a17a366d75123ec4e01fe7 Oct 31 15:36:10.164711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.164689 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66"] Oct 31 15:36:10.269617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.269583 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" event={"ID":"865c079d-7de8-4e62-98b6-4f06f5cef32e","Type":"ContainerStarted","Data":"dd6015fd706046ca848dad8dff06175ac87eae91bce648b4d68851aaddc33b18"} Oct 31 15:36:10.269759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.269649 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:10.269759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.269671 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" event={"ID":"865c079d-7de8-4e62-98b6-4f06f5cef32e","Type":"ContainerStarted","Data":"1e4912e84f35377f33f2d8aeed1518b05924ad1fd3a17a366d75123ec4e01fe7"} Oct 31 15:36:10.271388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.271362 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" event={"ID":"38ed24e4-2bc0-4510-acba-e13bcac34b49","Type":"ContainerStarted","Data":"28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34"} Oct 31 15:36:10.271545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.271406 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" event={"ID":"38ed24e4-2bc0-4510-acba-e13bcac34b49","Type":"ContainerStarted","Data":"4ef99425ceab0eb96d510ef1e8fcf94442965d85faf0d9705189ec6f2e8da2c7"} Oct 31 15:36:10.271545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.271433 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:10.391285 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.391237 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" podStartSLOduration=1.391223536 podStartE2EDuration="1.391223536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:36:10.387657508 +0000 UTC m=+3016.007208923" watchObservedRunningTime="2026-10-31 15:36:10.391223536 +0000 UTC m=+3016.010774941" Oct 31 15:36:10.508144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:10.507649 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" podStartSLOduration=1.507628753 podStartE2EDuration="1.507628753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:36:10.504078392 +0000 UTC m=+3016.123629805" watchObservedRunningTime="2026-10-31 15:36:10.507628753 +0000 UTC m=+3016.127180165" Oct 31 15:36:11.343418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:11.343382 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:16.028902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:16.028847 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:17.490601 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:36:17.490565 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:36:20.146363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:20.146315 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:36:20.774433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:20.774396 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:21.280972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:21.280942 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:36:21.281399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:21.281047 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:36:25.533900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:25.533857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:26.701974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.701942 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q"] Oct 31 15:36:26.706124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.706108 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.760869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.760841 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q"] Oct 31 15:36:26.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.804715 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.804767 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgsj\" (UniqueName: \"kubernetes.io/projected/0b59b90d-672d-4acc-93f3-2d238f9e4829-kube-api-access-tfgsj\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.804985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.804972 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.808678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.808651 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:36:26.808831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.808752 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:36:26.809034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.809013 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:36:26.809114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.809050 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:36:26.906390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.906351 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.906561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.906444 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.906561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.906479 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgsj\" (UniqueName: \"kubernetes.io/projected/0b59b90d-672d-4acc-93f3-2d238f9e4829-kube-api-access-tfgsj\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.907212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.907137 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.908831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.908812 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:26.964202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:26.964175 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgsj\" (UniqueName: \"kubernetes.io/projected/0b59b90d-672d-4acc-93f3-2d238f9e4829-kube-api-access-tfgsj\") pod \"maas-api-revoke-sub-d4ee73bf-2n47q\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:27.020143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:27.020119 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:27.396963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:27.396938 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q"] Oct 31 15:36:27.397440 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:36:27.397416 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b59b90d_672d_4acc_93f3_2d238f9e4829.slice/crio-08950dd6edb46f2a8c60a06a20749c67a63c7d7a70f2f0432a7a20698c5ab943 WatchSource:0}: Error finding container 08950dd6edb46f2a8c60a06a20749c67a63c7d7a70f2f0432a7a20698c5ab943: Status 404 returned error can't find the container with id 08950dd6edb46f2a8c60a06a20749c67a63c7d7a70f2f0432a7a20698c5ab943 Oct 31 15:36:28.396205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:28.396160 4865 generic.go:356] "Generic (PLEG): container finished" podID="0b59b90d-672d-4acc-93f3-2d238f9e4829" containerID="160d5ec1fc629801aff972b17faf8bd9c0cb414a9991bfa7f6d429cf14f8369c" exitCode=0 Oct 31 15:36:28.396718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:28.396348 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" event={"ID":"0b59b90d-672d-4acc-93f3-2d238f9e4829","Type":"ContainerDied","Data":"160d5ec1fc629801aff972b17faf8bd9c0cb414a9991bfa7f6d429cf14f8369c"} Oct 31 15:36:28.396718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:28.396447 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" event={"ID":"0b59b90d-672d-4acc-93f3-2d238f9e4829","Type":"ContainerStarted","Data":"08950dd6edb46f2a8c60a06a20749c67a63c7d7a70f2f0432a7a20698c5ab943"} Oct 31 15:36:29.567810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.567786 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:29.639516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.639452 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-xks-ca\") pod \"0b59b90d-672d-4acc-93f3-2d238f9e4829\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " Oct 31 15:36:29.639719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.639534 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-openshift-ca\") pod \"0b59b90d-672d-4acc-93f3-2d238f9e4829\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " Oct 31 15:36:29.639719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.639595 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b59b90d-672d-4acc-93f3-2d238f9e4829-kube-api-access-tfgsj\" (UniqueName: \"kubernetes.io/projected/0b59b90d-672d-4acc-93f3-2d238f9e4829-kube-api-access-tfgsj\") pod \"0b59b90d-672d-4acc-93f3-2d238f9e4829\" (UID: \"0b59b90d-672d-4acc-93f3-2d238f9e4829\") " Oct 31 15:36:29.639948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.639917 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-openshift-ca" pod "0b59b90d-672d-4acc-93f3-2d238f9e4829" (UID: "0b59b90d-672d-4acc-93f3-2d238f9e4829"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:36:29.640185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.640166 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:29.641965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.641936 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b59b90d-672d-4acc-93f3-2d238f9e4829-kube-api-access-tfgsj" pod "0b59b90d-672d-4acc-93f3-2d238f9e4829" (UID: "0b59b90d-672d-4acc-93f3-2d238f9e4829"). InnerVolumeSpecName "kube-api-access-tfgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:29.642103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.641980 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-xks-ca" pod "0b59b90d-672d-4acc-93f3-2d238f9e4829" (UID: "0b59b90d-672d-4acc-93f3-2d238f9e4829"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:36:29.741783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.741753 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b59b90d-672d-4acc-93f3-2d238f9e4829-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:29.741783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:29.741781 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfgsj\" (UniqueName: \"kubernetes.io/projected/0b59b90d-672d-4acc-93f3-2d238f9e4829-kube-api-access-tfgsj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:30.188086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:30.187990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:30.411314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:30.411283 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" Oct 31 15:36:30.411532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:30.411315 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q" event={"ID":"0b59b90d-672d-4acc-93f3-2d238f9e4829","Type":"ContainerDied","Data":"08950dd6edb46f2a8c60a06a20749c67a63c7d7a70f2f0432a7a20698c5ab943"} Oct 31 15:36:30.411532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:30.411361 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08950dd6edb46f2a8c60a06a20749c67a63c7d7a70f2f0432a7a20698c5ab943" Oct 31 15:36:34.898505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:34.898391 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:36.414837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:36.414772 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:36.827674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:36.827616 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:37.413658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.413608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:37.460608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.460577 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6"] Oct 31 15:36:37.461218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.461203 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0b59b90d-672d-4acc-93f3-2d238f9e4829" containerName="revoke-keys" Oct 31 15:36:37.461264 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.461219 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b59b90d-672d-4acc-93f3-2d238f9e4829" containerName="revoke-keys" Oct 31 15:36:37.461330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.461321 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b59b90d-672d-4acc-93f3-2d238f9e4829" containerName="revoke-keys" Oct 31 15:36:37.464985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.464969 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.534970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.534933 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6"] Oct 31 15:36:37.551541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.551507 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.551850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.551826 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.552140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.552120 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsxsv\" (UniqueName: \"kubernetes.io/projected/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-kube-api-access-nsxsv\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.569945 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.569913 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:36:37.570114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.570003 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:36:37.570226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.570205 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:36:37.583823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.583795 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:36:37.612607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.612569 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:37.652969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.652940 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsxsv\" (UniqueName: \"kubernetes.io/projected/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-kube-api-access-nsxsv\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.653162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.652984 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.653162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.653042 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.653657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.653627 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.655591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.655572 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.718200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.718167 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsxsv\" (UniqueName: \"kubernetes.io/projected/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-kube-api-access-nsxsv\") pod \"maas-api-revoke-sub-4683e84c-dzsv6\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:37.784421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:37.784392 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:38.158820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:38.158792 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6"] Oct 31 15:36:38.159815 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:36:38.159786 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6422ef93_38ad_49f9_8609_e6b0e88b9a4e.slice/crio-75990a17184ab9c031aef21001f657ae9b0b8ffcce1a4cb6628a47aa567ffb32 WatchSource:0}: Error finding container 75990a17184ab9c031aef21001f657ae9b0b8ffcce1a4cb6628a47aa567ffb32: Status 404 returned error can't find the container with id 75990a17184ab9c031aef21001f657ae9b0b8ffcce1a4cb6628a47aa567ffb32 Oct 31 15:36:38.463825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:38.463790 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" event={"ID":"6422ef93-38ad-49f9-8609-e6b0e88b9a4e","Type":"ContainerStarted","Data":"b67dba7f696c88e470559c8f960f72e796c7520c9f1f1c3096300d052076b89d"} Oct 31 15:36:38.464188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:38.463843 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" event={"ID":"6422ef93-38ad-49f9-8609-e6b0e88b9a4e","Type":"ContainerStarted","Data":"75990a17184ab9c031aef21001f657ae9b0b8ffcce1a4cb6628a47aa567ffb32"} Oct 31 15:36:38.580410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:38.580357 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" podStartSLOduration=1.580344981 podStartE2EDuration="1.580344981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:36:38.578502359 +0000 UTC m=+3044.198053771" watchObservedRunningTime="2026-10-31 15:36:38.580344981 +0000 UTC m=+3044.199896415" Oct 31 15:36:39.472965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:39.472934 4865 generic.go:356] "Generic (PLEG): container finished" podID="6422ef93-38ad-49f9-8609-e6b0e88b9a4e" containerID="b67dba7f696c88e470559c8f960f72e796c7520c9f1f1c3096300d052076b89d" exitCode=0 Oct 31 15:36:39.473393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:39.473031 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" event={"ID":"6422ef93-38ad-49f9-8609-e6b0e88b9a4e","Type":"ContainerDied","Data":"b67dba7f696c88e470559c8f960f72e796c7520c9f1f1c3096300d052076b89d"} Oct 31 15:36:40.035722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.035688 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:40.643382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.643357 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:40.691440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.691411 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-kube-api-access-nsxsv\" (UniqueName: \"kubernetes.io/projected/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-kube-api-access-nsxsv\") pod \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " Oct 31 15:36:40.691613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.691516 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-openshift-ca\") pod \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " Oct 31 15:36:40.691613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.691577 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-xks-ca\") pod \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\" (UID: \"6422ef93-38ad-49f9-8609-e6b0e88b9a4e\") " Oct 31 15:36:40.691847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.691825 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-openshift-ca" pod "6422ef93-38ad-49f9-8609-e6b0e88b9a4e" (UID: "6422ef93-38ad-49f9-8609-e6b0e88b9a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:36:40.692223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.692151 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:40.693709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.693684 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-xks-ca" pod "6422ef93-38ad-49f9-8609-e6b0e88b9a4e" (UID: "6422ef93-38ad-49f9-8609-e6b0e88b9a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:36:40.693821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.693792 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-kube-api-access-nsxsv" pod "6422ef93-38ad-49f9-8609-e6b0e88b9a4e" (UID: "6422ef93-38ad-49f9-8609-e6b0e88b9a4e"). InnerVolumeSpecName "kube-api-access-nsxsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:40.793957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.793878 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsxsv\" (UniqueName: \"kubernetes.io/projected/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-kube-api-access-nsxsv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:40.793957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:40.793909 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6422ef93-38ad-49f9-8609-e6b0e88b9a4e-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:41.460712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:41.460678 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:41.487851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:41.487824 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" Oct 31 15:36:41.488010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:41.487824 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6" event={"ID":"6422ef93-38ad-49f9-8609-e6b0e88b9a4e","Type":"ContainerDied","Data":"75990a17184ab9c031aef21001f657ae9b0b8ffcce1a4cb6628a47aa567ffb32"} Oct 31 15:36:41.488010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:41.487958 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75990a17184ab9c031aef21001f657ae9b0b8ffcce1a4cb6628a47aa567ffb32" Oct 31 15:36:41.636569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:41.636532 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:41.773626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:41.773547 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:44.673589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:44.673557 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:44.849975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:44.849943 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:45.100272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.100238 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj"] Oct 31 15:36:45.100984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.100966 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6422ef93-38ad-49f9-8609-e6b0e88b9a4e" containerName="revoke-keys" Oct 31 15:36:45.101044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.100986 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6422ef93-38ad-49f9-8609-e6b0e88b9a4e" containerName="revoke-keys" Oct 31 15:36:45.101104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.101094 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6422ef93-38ad-49f9-8609-e6b0e88b9a4e" containerName="revoke-keys" Oct 31 15:36:45.104752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.104733 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.160229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.160201 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj"] Oct 31 15:36:45.204708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.204675 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c947d9\"" Oct 31 15:36:45.210349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.210298 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:45.246844 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.246806 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb2281f-b896-4ecb-bb3b-876a10f47803-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.247048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.246923 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qskww\" (UniqueName: \"kubernetes.io/projected/feb2281f-b896-4ecb-bb3b-876a10f47803-kube-api-access-qskww\") pod \"payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.248476 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.248438 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6"] Oct 31 15:36:45.260901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.260859 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.323244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.323188 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c947d9-dockercfg-rbnsl\"" Oct 31 15:36:45.336668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.336632 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6"] Oct 31 15:36:45.354458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.354376 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb2281f-b896-4ecb-bb3b-876a10f47803-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.354744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.354705 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qskww\" (UniqueName: \"kubernetes.io/projected/feb2281f-b896-4ecb-bb3b-876a10f47803-kube-api-access-qskww\") pod \"payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.354856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.354777 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gn4\" (UniqueName: \"kubernetes.io/projected/28a82de8-9d25-4b34-8edc-b03ab1e1b689-kube-api-access-p5gn4\") pod \"payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.354856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.354838 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a82de8-9d25-4b34-8edc-b03ab1e1b689-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.356144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.356120 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb2281f-b896-4ecb-bb3b-876a10f47803-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.414268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.414232 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qskww\" (UniqueName: \"kubernetes.io/projected/feb2281f-b896-4ecb-bb3b-876a10f47803-kube-api-access-qskww\") pod \"payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.419224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.419199 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:45.456210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.456176 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gn4\" (UniqueName: \"kubernetes.io/projected/28a82de8-9d25-4b34-8edc-b03ab1e1b689-kube-api-access-p5gn4\") pod \"payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.456377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.456230 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a82de8-9d25-4b34-8edc-b03ab1e1b689-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.457029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.457006 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a82de8-9d25-4b34-8edc-b03ab1e1b689-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.517180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.517147 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gn4\" (UniqueName: \"kubernetes.io/projected/28a82de8-9d25-4b34-8edc-b03ab1e1b689-kube-api-access-p5gn4\") pod \"payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.630507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.630395 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:45.801380 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:36:45.801348 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb2281f_b896_4ecb_bb3b_876a10f47803.slice/crio-a872e6306082fad040c61fac3011fd8ca410bafbce9237ee4fe3cd074df1f7a5 WatchSource:0}: Error finding container a872e6306082fad040c61fac3011fd8ca410bafbce9237ee4fe3cd074df1f7a5: Status 404 returned error can't find the container with id a872e6306082fad040c61fac3011fd8ca410bafbce9237ee4fe3cd074df1f7a5 Oct 31 15:36:45.802001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:45.801975 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj"] Oct 31 15:36:46.019216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.019182 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6"] Oct 31 15:36:46.019343 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:36:46.019191 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a82de8_9d25_4b34_8edc_b03ab1e1b689.slice/crio-7e2d7e4fd316d96a4ba9a0c2ead8a199b5d27f25daeed3e13de1e78c7bdc2a82 WatchSource:0}: Error finding container 7e2d7e4fd316d96a4ba9a0c2ead8a199b5d27f25daeed3e13de1e78c7bdc2a82: Status 404 returned error can't find the container with id 7e2d7e4fd316d96a4ba9a0c2ead8a199b5d27f25daeed3e13de1e78c7bdc2a82 Oct 31 15:36:46.523943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.523908 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" event={"ID":"28a82de8-9d25-4b34-8edc-b03ab1e1b689","Type":"ContainerStarted","Data":"e1d913ea8d192ab658cfff55f3387921bb8a236dc50a302804bdc3cbbb1a49ac"} Oct 31 15:36:46.524156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.523970 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" event={"ID":"28a82de8-9d25-4b34-8edc-b03ab1e1b689","Type":"ContainerStarted","Data":"7e2d7e4fd316d96a4ba9a0c2ead8a199b5d27f25daeed3e13de1e78c7bdc2a82"} Oct 31 15:36:46.524156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.523999 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:46.525884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.525854 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" event={"ID":"feb2281f-b896-4ecb-bb3b-876a10f47803","Type":"ContainerStarted","Data":"98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5"} Oct 31 15:36:46.526029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.525905 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" event={"ID":"feb2281f-b896-4ecb-bb3b-876a10f47803","Type":"ContainerStarted","Data":"a872e6306082fad040c61fac3011fd8ca410bafbce9237ee4fe3cd074df1f7a5"} Oct 31 15:36:46.526029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.525935 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:46.643133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.643085 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" podStartSLOduration=1.6430704839999999 podStartE2EDuration="1.643070484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:36:46.639260845 +0000 UTC m=+3052.258812257" watchObservedRunningTime="2026-10-31 15:36:46.643070484 +0000 UTC m=+3052.262621939" Oct 31 15:36:46.762175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:46.762119 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" podStartSLOduration=1.762104506 podStartE2EDuration="1.762104506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:36:46.757499668 +0000 UTC m=+3052.377051074" watchObservedRunningTime="2026-10-31 15:36:46.762104506 +0000 UTC m=+3052.381655916" Oct 31 15:36:47.490442 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:36:47.490410 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:36:49.894935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:49.894899 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:51.318941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:51.318903 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:51.319441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:51.319423 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq"] Oct 31 15:36:54.525120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:54.525065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:55.791965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:55.791932 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:36:57.536998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:57.536970 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 15:36:57.537401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:57.537042 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 15:36:59.295093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:36:59.295062 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:02.122842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:02.122809 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:37:02.412676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:02.412592 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:02.741987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:02.741951 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:02.863106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:02.863072 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:03.847507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:03.847459 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:03.969132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:03.969100 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:06.340819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.340783 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" podUID="bc4bcff4-fa4d-4420-aca1-748ff5e8181c" containerName="main" containerID="cri-o://e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5" gracePeriod=45 Oct 31 15:37:06.641149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.641123 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:37:06.679627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.679596 4865 generic.go:356] "Generic (PLEG): container finished" podID="bc4bcff4-fa4d-4420-aca1-748ff5e8181c" containerID="e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5" exitCode=0 Oct 31 15:37:06.679837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.679633 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" event={"ID":"bc4bcff4-fa4d-4420-aca1-748ff5e8181c","Type":"ContainerDied","Data":"e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5"} Oct 31 15:37:06.679837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.679689 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" event={"ID":"bc4bcff4-fa4d-4420-aca1-748ff5e8181c","Type":"ContainerDied","Data":"9024796fa53be4f3078dbce64091377981141c5076eb6781410417631f049f0a"} Oct 31 15:37:06.679837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.679711 4865 scope.go:122] "RemoveContainer" containerID="e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5" Oct 31 15:37:06.679837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.679650 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" Oct 31 15:37:06.691298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.691275 4865 scope.go:122] "RemoveContainer" containerID="e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5" Oct 31 15:37:06.691623 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:37:06.691602 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5\": container with ID starting with e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5 not found: ID does not exist" containerID="e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5" Oct 31 15:37:06.691696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.691632 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5"} err="failed to get container status \"e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5\": rpc error: code = NotFound desc = could not find container \"e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5\": container with ID starting with e7b1b6d27048a5c37e3c6c2a2a1e8a756130edf29f0658df1bbf8a418e762ce5 not found: ID does not exist" Oct 31 15:37:06.748845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.748816 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-dshm\") pod \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " Oct 31 15:37:06.748845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.748850 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tmp-dir\") pod \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " Oct 31 15:37:06.749043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.748989 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-model-cache\") pod \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " Oct 31 15:37:06.749149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749124 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tmp-dir" pod "bc4bcff4-fa4d-4420-aca1-748ff5e8181c" (UID: "bc4bcff4-fa4d-4420-aca1-748ff5e8181c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:06.749204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749133 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tls-certs\") pod \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " Oct 31 15:37:06.749254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749223 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-model-cache" pod "bc4bcff4-fa4d-4420-aca1-748ff5e8181c" (UID: "bc4bcff4-fa4d-4420-aca1-748ff5e8181c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:06.749254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749229 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-kube-api-access-6crgr\" (UniqueName: \"kubernetes.io/projected/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-kube-api-access-6crgr\") pod \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " Oct 31 15:37:06.749376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749293 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-home\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-home\") pod \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\" (UID: \"bc4bcff4-fa4d-4420-aca1-748ff5e8181c\") " Oct 31 15:37:06.749631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749605 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-home" pod "bc4bcff4-fa4d-4420-aca1-748ff5e8181c" (UID: "bc4bcff4-fa4d-4420-aca1-748ff5e8181c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:06.749923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749896 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:06.749923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749914 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:06.750059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.749929 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:06.751537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.751509 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tls-certs" pod "bc4bcff4-fa4d-4420-aca1-748ff5e8181c" (UID: "bc4bcff4-fa4d-4420-aca1-748ff5e8181c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:37:06.751791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.751774 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-dshm" pod "bc4bcff4-fa4d-4420-aca1-748ff5e8181c" (UID: "bc4bcff4-fa4d-4420-aca1-748ff5e8181c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:06.751880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.751859 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-kube-api-access-6crgr" pod "bc4bcff4-fa4d-4420-aca1-748ff5e8181c" (UID: "bc4bcff4-fa4d-4420-aca1-748ff5e8181c"). InnerVolumeSpecName "kube-api-access-6crgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:06.851388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.851351 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:06.851388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.851384 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6crgr\" (UniqueName: \"kubernetes.io/projected/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-kube-api-access-6crgr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:06.851388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:06.851408 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4bcff4-fa4d-4420-aca1-748ff5e8181c-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:07.102229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:07.102199 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq"] Oct 31 15:37:07.105605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:07.105582 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq"] Oct 31 15:37:07.621241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:07.621205 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-531c3c/auth-test-model-531c3c-kserve-54fffd4d4f-j9tsq" podUID="bc4bcff4-fa4d-4420-aca1-748ff5e8181c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.26:8000/ready\": context deadline exceeded" Oct 31 15:37:08.497451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:08.497417 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4bcff4-fa4d-4420-aca1-748ff5e8181c" path="/var/lib/kubelet/pods/bc4bcff4-fa4d-4420-aca1-748ff5e8181c/volumes" Oct 31 15:37:08.606736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:08.606696 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:08.727994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:08.727962 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:08.849703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:08.849591 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:13.389026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:13.388973 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:17.490923 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:37:17.490883 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:37:18.091740 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:18.091696 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:22.724528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:22.724462 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:27.411175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:27.411097 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:32.244172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:32.244132 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:36.365070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:36.365031 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6"] Oct 31 15:37:36.384252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:36.384207 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:37.006991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:37.006956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:38.687101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:38.687066 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:39.277382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.277342 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5"] Oct 31 15:37:39.278347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.278325 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bc4bcff4-fa4d-4420-aca1-748ff5e8181c" containerName="main" Oct 31 15:37:39.278413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.278350 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4bcff4-fa4d-4420-aca1-748ff5e8181c" containerName="main" Oct 31 15:37:39.278577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.278565 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4bcff4-fa4d-4420-aca1-748ff5e8181c" containerName="main" Oct 31 15:37:39.282030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.282015 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.361838 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwr6v\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-kube-api-access-xwr6v\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.361991 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-token\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.362050 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-credential-socket\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.362078 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istiod-ca-cert\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.362134 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-data\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.362199 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-envoy\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.362247 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-certs\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.362318 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-socket\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.362639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.362374 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-podinfo\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.378358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.378169 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:39.395409 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.395375 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-982f43-openshift-default-dockercfg-8z4g8\"" Oct 31 15:37:39.397027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.397002 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5"] Oct 31 15:37:39.464617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.464504 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-credential-socket\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.464871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.464845 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istiod-ca-cert\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.464968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.464920 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-data\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.464968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.464944 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-credential-socket\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.464986 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-envoy\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.465035 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-certs\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.465096 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-socket\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.465153 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-podinfo\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.465206 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwr6v\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-kube-api-access-xwr6v\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.465344 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-token\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.465721 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istiod-ca-cert\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.465986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.465967 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-data\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.466152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.466111 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-socket\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.466790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.466767 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-certs\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.472346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.471671 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-podinfo\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.473990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.473965 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-envoy\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.523903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.523870 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-token\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.524071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.524038 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwr6v\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-kube-api-access-xwr6v\") pod \"e2e-extmod-a-982f43-openshift-default-66f447d649-659h5\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.595425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.595338 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:39.841926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:39.841890 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:40.031780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.031741 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5"] Oct 31 15:37:40.031905 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:37:40.031820 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c868694_55bc_4cb9_83cb_03e07dfe35ba.slice/crio-0784214c097bc81572f85fae3a22eb86761fcb888e1e63ee24dee693cbe2baa7 WatchSource:0}: Error finding container 0784214c097bc81572f85fae3a22eb86761fcb888e1e63ee24dee693cbe2baa7: Status 404 returned error can't find the container with id 0784214c097bc81572f85fae3a22eb86761fcb888e1e63ee24dee693cbe2baa7 Oct 31 15:37:40.034614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.034579 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:37:40.034720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.034643 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:37:40.034720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.034675 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:37:40.437195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.437120 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd"] Oct 31 15:37:40.441319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.441300 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.495667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.495637 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:37:40.495667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.495641 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:37:40.495897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.495723 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:37:40.495897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.495815 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:37:40.500024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.499996 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd"] Oct 31 15:37:40.590982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.590949 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.591156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.591111 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.591255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.591236 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w7m\" (UniqueName: \"kubernetes.io/projected/4a2241d3-0dcf-4ae2-924b-f654554ea568-kube-api-access-x4w7m\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.692919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.692828 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.693098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.692936 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w7m\" (UniqueName: \"kubernetes.io/projected/4a2241d3-0dcf-4ae2-924b-f654554ea568-kube-api-access-x4w7m\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.693098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.693054 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.693590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.693568 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.695801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.695776 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.750876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.750841 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w7m\" (UniqueName: \"kubernetes.io/projected/4a2241d3-0dcf-4ae2-924b-f654554ea568-kube-api-access-x4w7m\") pod \"maas-api-revoke-sub-289b757e-cclpd\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.759682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.759658 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:40.915424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.915376 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" event={"ID":"2c868694-55bc-4cb9-83cb-03e07dfe35ba","Type":"ContainerStarted","Data":"5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424"} Oct 31 15:37:40.915424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:40.915428 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" event={"ID":"2c868694-55bc-4cb9-83cb-03e07dfe35ba","Type":"ContainerStarted","Data":"0784214c097bc81572f85fae3a22eb86761fcb888e1e63ee24dee693cbe2baa7"} Oct 31 15:37:41.033031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.032974 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podStartSLOduration=2.032960802 podStartE2EDuration="2.032960802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:41.031841031 +0000 UTC m=+3106.651392445" watchObservedRunningTime="2026-10-31 15:37:41.032960802 +0000 UTC m=+3106.652512294" Oct 31 15:37:41.063732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.063693 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r"] Oct 31 15:37:41.063952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.063930 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" containerID="cri-o://64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6" gracePeriod=30 Oct 31 15:37:41.144708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.144678 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd"] Oct 31 15:37:41.145290 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:37:41.145257 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2241d3_0dcf_4ae2_924b_f654554ea568.slice/crio-74b6b4f618898bb5b854bb804c82cad7b97bdd083bd2517533b122267e553344 WatchSource:0}: Error finding container 74b6b4f618898bb5b854bb804c82cad7b97bdd083bd2517533b122267e553344: Status 404 returned error can't find the container with id 74b6b4f618898bb5b854bb804c82cad7b97bdd083bd2517533b122267e553344 Oct 31 15:37:41.595788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.595756 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:41.601508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.601453 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:41.667628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.667543 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz"] Oct 31 15:37:41.671753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.671735 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.730078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.730041 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz"] Oct 31 15:37:41.806777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.806737 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.806986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.806951 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.807053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.807016 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdxc\" (UniqueName: \"kubernetes.io/projected/ce51ff42-0493-447d-9401-89804bbe8115-kube-api-access-qsdxc\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.908129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.908090 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.908335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.908169 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdxc\" (UniqueName: \"kubernetes.io/projected/ce51ff42-0493-447d-9401-89804bbe8115-kube-api-access-qsdxc\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.908335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.908242 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.908772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.908753 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.910212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.910187 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:41.910930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.910911 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.925296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.925205 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" event={"ID":"4a2241d3-0dcf-4ae2-924b-f654554ea568","Type":"ContainerStarted","Data":"e6854fbdf7bdf12d186b3c0f6cddd83468ca8207312c6704e74c0eceb049e7d8"} Oct 31 15:37:41.925296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.925269 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" event={"ID":"4a2241d3-0dcf-4ae2-924b-f654554ea568","Type":"ContainerStarted","Data":"74b6b4f618898bb5b854bb804c82cad7b97bdd083bd2517533b122267e553344"} Oct 31 15:37:41.925778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.925758 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:41.926821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.926802 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:37:41.983252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.983214 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdxc\" (UniqueName: \"kubernetes.io/projected/ce51ff42-0493-447d-9401-89804bbe8115-kube-api-access-qsdxc\") pod \"maas-api-revoke-sub-0d4b7f1a-sdxvz\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:41.989085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:41.989055 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:42.042456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:42.042397 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" podStartSLOduration=2.04237934 podStartE2EDuration="2.04237934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:42.039910741 +0000 UTC m=+3107.659462146" watchObservedRunningTime="2026-10-31 15:37:42.04237934 +0000 UTC m=+3107.661930750" Oct 31 15:37:42.373183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:42.373147 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz"] Oct 31 15:37:42.933586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:42.933545 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" event={"ID":"ce51ff42-0493-447d-9401-89804bbe8115","Type":"ContainerStarted","Data":"addf73dd43da7f71d1f9a7fabde3865627c19c3fb489b35367c8ca0a2352b9f1"} Oct 31 15:37:42.934092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:42.933611 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" event={"ID":"ce51ff42-0493-447d-9401-89804bbe8115","Type":"ContainerStarted","Data":"50c6225252a95be3976ababacc42d21f0ddb974101ebafc89eb31d821ada0148"} Oct 31 15:37:42.935196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:42.935169 4865 generic.go:356] "Generic (PLEG): container finished" podID="4a2241d3-0dcf-4ae2-924b-f654554ea568" containerID="e6854fbdf7bdf12d186b3c0f6cddd83468ca8207312c6704e74c0eceb049e7d8" exitCode=0 Oct 31 15:37:42.935346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:42.935250 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" event={"ID":"4a2241d3-0dcf-4ae2-924b-f654554ea568","Type":"ContainerDied","Data":"e6854fbdf7bdf12d186b3c0f6cddd83468ca8207312c6704e74c0eceb049e7d8"} Oct 31 15:37:43.049049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:43.048987 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" podStartSLOduration=2.048973717 podStartE2EDuration="2.048973717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:43.047602175 +0000 UTC m=+3108.667153585" watchObservedRunningTime="2026-10-31 15:37:43.048973717 +0000 UTC m=+3108.668525127" Oct 31 15:37:43.944693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:43.944661 4865 generic.go:356] "Generic (PLEG): container finished" podID="ce51ff42-0493-447d-9401-89804bbe8115" containerID="addf73dd43da7f71d1f9a7fabde3865627c19c3fb489b35367c8ca0a2352b9f1" exitCode=0 Oct 31 15:37:43.945171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:43.944745 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" event={"ID":"ce51ff42-0493-447d-9401-89804bbe8115","Type":"ContainerDied","Data":"addf73dd43da7f71d1f9a7fabde3865627c19c3fb489b35367c8ca0a2352b9f1"} Oct 31 15:37:44.107532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.107503 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:44.239576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.239541 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-xks-ca\") pod \"4a2241d3-0dcf-4ae2-924b-f654554ea568\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " Oct 31 15:37:44.239763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.239596 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2241d3-0dcf-4ae2-924b-f654554ea568-kube-api-access-x4w7m\" (UniqueName: \"kubernetes.io/projected/4a2241d3-0dcf-4ae2-924b-f654554ea568-kube-api-access-x4w7m\") pod \"4a2241d3-0dcf-4ae2-924b-f654554ea568\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " Oct 31 15:37:44.239763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.239650 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-openshift-ca\") pod \"4a2241d3-0dcf-4ae2-924b-f654554ea568\" (UID: \"4a2241d3-0dcf-4ae2-924b-f654554ea568\") " Oct 31 15:37:44.240143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.240114 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-openshift-ca" pod "4a2241d3-0dcf-4ae2-924b-f654554ea568" (UID: "4a2241d3-0dcf-4ae2-924b-f654554ea568"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:37:44.240309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.240289 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:44.241987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.241959 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2241d3-0dcf-4ae2-924b-f654554ea568-kube-api-access-x4w7m" pod "4a2241d3-0dcf-4ae2-924b-f654554ea568" (UID: "4a2241d3-0dcf-4ae2-924b-f654554ea568"). InnerVolumeSpecName "kube-api-access-x4w7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:44.242315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.242292 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-xks-ca" pod "4a2241d3-0dcf-4ae2-924b-f654554ea568" (UID: "4a2241d3-0dcf-4ae2-924b-f654554ea568"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:37:44.341104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.341074 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2241d3-0dcf-4ae2-924b-f654554ea568-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:44.341104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.341103 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4w7m\" (UniqueName: \"kubernetes.io/projected/4a2241d3-0dcf-4ae2-924b-f654554ea568-kube-api-access-x4w7m\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:44.493487 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.493382 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:44.663617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.663579 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:44.794340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.794238 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:44.953345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.953309 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" Oct 31 15:37:44.953860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.953344 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd" event={"ID":"4a2241d3-0dcf-4ae2-924b-f654554ea568","Type":"ContainerDied","Data":"74b6b4f618898bb5b854bb804c82cad7b97bdd083bd2517533b122267e553344"} Oct 31 15:37:44.953860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:44.953393 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74b6b4f618898bb5b854bb804c82cad7b97bdd083bd2517533b122267e553344" Oct 31 15:37:45.069353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.069267 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-4n56r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:37:45.069353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.069330 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:37:45.110562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.110536 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:45.255001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.254962 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-openshift-ca\") pod \"ce51ff42-0493-447d-9401-89804bbe8115\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " Oct 31 15:37:45.255193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.255152 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce51ff42-0493-447d-9401-89804bbe8115-kube-api-access-qsdxc\" (UniqueName: \"kubernetes.io/projected/ce51ff42-0493-447d-9401-89804bbe8115-kube-api-access-qsdxc\") pod \"ce51ff42-0493-447d-9401-89804bbe8115\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " Oct 31 15:37:45.255252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.255202 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-xks-ca\") pod \"ce51ff42-0493-447d-9401-89804bbe8115\" (UID: \"ce51ff42-0493-447d-9401-89804bbe8115\") " Oct 31 15:37:45.255367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.255341 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-openshift-ca" pod "ce51ff42-0493-447d-9401-89804bbe8115" (UID: "ce51ff42-0493-447d-9401-89804bbe8115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:37:45.255841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.255778 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:45.257594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.257568 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-xks-ca" pod "ce51ff42-0493-447d-9401-89804bbe8115" (UID: "ce51ff42-0493-447d-9401-89804bbe8115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:37:45.257701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.257614 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce51ff42-0493-447d-9401-89804bbe8115-kube-api-access-qsdxc" pod "ce51ff42-0493-447d-9401-89804bbe8115" (UID: "ce51ff42-0493-447d-9401-89804bbe8115"). InnerVolumeSpecName "kube-api-access-qsdxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:45.357528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.357407 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsdxc\" (UniqueName: \"kubernetes.io/projected/ce51ff42-0493-447d-9401-89804bbe8115-kube-api-access-qsdxc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:45.357528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.357441 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce51ff42-0493-447d-9401-89804bbe8115-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:45.961710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.961679 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" Oct 31 15:37:45.962094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.961679 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz" event={"ID":"ce51ff42-0493-447d-9401-89804bbe8115","Type":"ContainerDied","Data":"50c6225252a95be3976ababacc42d21f0ddb974101ebafc89eb31d821ada0148"} Oct 31 15:37:45.962094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:45.961838 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c6225252a95be3976ababacc42d21f0ddb974101ebafc89eb31d821ada0148" Oct 31 15:37:46.347532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.347505 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:37:46.472044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472008 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-certs\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472058 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istiod-ca-cert\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472079 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-envoy\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472118 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-credential-socket\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472249 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472204 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-token\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-token\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472275 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-kube-api-access-njggk\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-kube-api-access-njggk\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472302 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-podinfo\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472352 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-data\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472351 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-certs" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:46.472527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472435 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-socket\") pod \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\" (UID: \"af2bf269-a9df-49d5-a4e2-ba7efdabf735\") " Oct 31 15:37:46.472800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472543 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istiod-ca-cert" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:37:46.472800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472603 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-credential-socket" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:46.472800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472779 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-data" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:46.472947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.472820 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-socket" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:46.473041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.473021 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.473088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.473047 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.473088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.473062 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.473088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.473076 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.473230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.473090 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.474780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.474747 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-envoy" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:46.475312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.475293 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-podinfo" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:37:46.475813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.475570 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-token" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:46.475813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.475694 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-kube-api-access-njggk" pod "af2bf269-a9df-49d5-a4e2-ba7efdabf735" (UID: "af2bf269-a9df-49d5-a4e2-ba7efdabf735"). InnerVolumeSpecName "kube-api-access-njggk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:46.574326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.574288 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.574326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.574321 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njggk\" (UniqueName: \"kubernetes.io/projected/af2bf269-a9df-49d5-a4e2-ba7efdabf735-kube-api-access-njggk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.574326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.574334 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.574626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.574347 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af2bf269-a9df-49d5-a4e2-ba7efdabf735-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:46.865269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.865188 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:46.970014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.969983 4865 generic.go:356] "Generic (PLEG): container finished" podID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerID="64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6" exitCode=0 Oct 31 15:37:46.970508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.970034 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" Oct 31 15:37:46.970508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.970063 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" event={"ID":"af2bf269-a9df-49d5-a4e2-ba7efdabf735","Type":"ContainerDied","Data":"64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6"} Oct 31 15:37:46.970508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.970128 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r" event={"ID":"af2bf269-a9df-49d5-a4e2-ba7efdabf735","Type":"ContainerDied","Data":"26cc3c988b74f2310c6f0342c06968b11162f44390e6a7fcbe1715647c28e82b"} Oct 31 15:37:46.970508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.970151 4865 scope.go:122] "RemoveContainer" containerID="64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6" Oct 31 15:37:46.980957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.980934 4865 scope.go:122] "RemoveContainer" containerID="64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6" Oct 31 15:37:46.981232 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:37:46.981213 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6\": container with ID starting with 64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6 not found: ID does not exist" containerID="64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6" Oct 31 15:37:46.981295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:46.981242 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6"} err="failed to get container status \"64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6\": rpc error: code = NotFound desc = could not find container \"64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6\": container with ID starting with 64ae3707995c7fc26e80a89055985ca0d7a2e6d7d47f50874daac29e1d5662e6 not found: ID does not exist" Oct 31 15:37:47.017898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:47.017871 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:47.140993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:47.140910 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r"] Oct 31 15:37:47.145696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:47.145672 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-4n56r"] Oct 31 15:37:47.225903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:47.225871 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:47.491761 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:37:47.491728 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:37:48.497028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:48.496996 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" path="/var/lib/kubelet/pods/af2bf269-a9df-49d5-a4e2-ba7efdabf735/volumes" Oct 31 15:37:50.805939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:50.805899 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:50.981424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:50.981364 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:51.185448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.185362 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl"] Oct 31 15:37:51.186086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186068 4865 cpu_manager.go:414] "containerMap: removing container" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" Oct 31 15:37:51.186141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186089 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" Oct 31 15:37:51.186141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186104 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4a2241d3-0dcf-4ae2-924b-f654554ea568" containerName="revoke-keys" Oct 31 15:37:51.186141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186110 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2241d3-0dcf-4ae2-924b-f654554ea568" containerName="revoke-keys" Oct 31 15:37:51.186141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186128 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ce51ff42-0493-447d-9401-89804bbe8115" containerName="revoke-keys" Oct 31 15:37:51.186141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186133 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce51ff42-0493-447d-9401-89804bbe8115" containerName="revoke-keys" Oct 31 15:37:51.186339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186225 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2bf269-a9df-49d5-a4e2-ba7efdabf735" containerName="istio-proxy" Oct 31 15:37:51.186339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186234 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2241d3-0dcf-4ae2-924b-f654554ea568" containerName="revoke-keys" Oct 31 15:37:51.186339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.186242 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce51ff42-0493-447d-9401-89804bbe8115" containerName="revoke-keys" Oct 31 15:37:51.191199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.191184 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.222042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.222017 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc"] Oct 31 15:37:51.226242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.226216 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.243868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.243841 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl"] Oct 31 15:37:51.248932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.248902 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94q7b\" (UniqueName: \"kubernetes.io/projected/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-kube-api-access-94q7b\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.249094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.249076 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.249169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.249131 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.269089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.269066 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:37:51.269089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.269084 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:37:51.269281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.269098 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:37:51.291747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.291718 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:37:51.296509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.296459 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc"] Oct 31 15:37:51.346929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.346893 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-982f43-dockercfg-jp85b\"" Oct 31 15:37:51.348322 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.348293 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-982f43\"" Oct 31 15:37:51.360561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.351101 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.360561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.351197 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.360561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.351275 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwctp\" (UniqueName: \"kubernetes.io/projected/48c77f43-fb3e-4376-9454-a8cbede3f463-kube-api-access-wwctp\") pod \"payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.360561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.351439 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c77f43-fb3e-4376-9454-a8cbede3f463-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.360561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.351563 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94q7b\" (UniqueName: \"kubernetes.io/projected/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-kube-api-access-94q7b\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.360561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.352732 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.371379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.364093 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.378907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.378874 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj"] Oct 31 15:37:51.384358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.384324 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.391785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.391750 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" podUID="b80e12f0-24c8-4b17-8201-1d6d5b259771" containerName="main" containerID="cri-o://d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4" gracePeriod=45 Oct 31 15:37:51.410518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.409861 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94q7b\" (UniqueName: \"kubernetes.io/projected/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-kube-api-access-94q7b\") pod \"maas-api-revoke-sub-ed06859d-cbtjl\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.434764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.434737 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj"] Oct 31 15:37:51.452786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.452715 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwctp\" (UniqueName: \"kubernetes.io/projected/48c77f43-fb3e-4376-9454-a8cbede3f463-kube-api-access-wwctp\") pod \"payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.452922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.452813 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c77f43-fb3e-4376-9454-a8cbede3f463-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.452977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.452939 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65cxn\" (UniqueName: \"kubernetes.io/projected/fc9843c1-ac33-4300-a67c-368c336625d3-kube-api-access-65cxn\") pod \"payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.453090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.453065 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9843c1-ac33-4300-a67c-368c336625d3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.453596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.453576 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c77f43-fb3e-4376-9454-a8cbede3f463-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.506044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.506013 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:51.510400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.510377 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwctp\" (UniqueName: \"kubernetes.io/projected/48c77f43-fb3e-4376-9454-a8cbede3f463-kube-api-access-wwctp\") pod \"payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.540980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.540951 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:51.555387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.555336 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65cxn\" (UniqueName: \"kubernetes.io/projected/fc9843c1-ac33-4300-a67c-368c336625d3-kube-api-access-65cxn\") pod \"payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.555559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.555504 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9843c1-ac33-4300-a67c-368c336625d3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.557225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.557200 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9843c1-ac33-4300-a67c-368c336625d3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.614021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.613982 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65cxn\" (UniqueName: \"kubernetes.io/projected/fc9843c1-ac33-4300-a67c-368c336625d3-kube-api-access-65cxn\") pod \"payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.700864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.700647 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:51.730917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.730893 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:37:51.758439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758405 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-model-cache\") pod \"b80e12f0-24c8-4b17-8201-1d6d5b259771\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " Oct 31 15:37:51.758686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758565 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80e12f0-24c8-4b17-8201-1d6d5b259771-tls-certs\" (UniqueName: \"kubernetes.io/secret/b80e12f0-24c8-4b17-8201-1d6d5b259771-tls-certs\") pod \"b80e12f0-24c8-4b17-8201-1d6d5b259771\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " Oct 31 15:37:51.758686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758600 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-dshm\") pod \"b80e12f0-24c8-4b17-8201-1d6d5b259771\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " Oct 31 15:37:51.758686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758665 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-tmp-dir\") pod \"b80e12f0-24c8-4b17-8201-1d6d5b259771\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " Oct 31 15:37:51.758871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758699 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-model-cache" pod "b80e12f0-24c8-4b17-8201-1d6d5b259771" (UID: "b80e12f0-24c8-4b17-8201-1d6d5b259771"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:51.758871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758726 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-home\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-home\") pod \"b80e12f0-24c8-4b17-8201-1d6d5b259771\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " Oct 31 15:37:51.758871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758792 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80e12f0-24c8-4b17-8201-1d6d5b259771-kube-api-access-lqdts\" (UniqueName: \"kubernetes.io/projected/b80e12f0-24c8-4b17-8201-1d6d5b259771-kube-api-access-lqdts\") pod \"b80e12f0-24c8-4b17-8201-1d6d5b259771\" (UID: \"b80e12f0-24c8-4b17-8201-1d6d5b259771\") " Oct 31 15:37:51.759054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.758910 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-tmp-dir" pod "b80e12f0-24c8-4b17-8201-1d6d5b259771" (UID: "b80e12f0-24c8-4b17-8201-1d6d5b259771"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:51.759054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.759044 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-home" pod "b80e12f0-24c8-4b17-8201-1d6d5b259771" (UID: "b80e12f0-24c8-4b17-8201-1d6d5b259771"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:51.759590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.759488 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:51.759590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.759539 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:51.759590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.759553 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:51.761776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.761740 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80e12f0-24c8-4b17-8201-1d6d5b259771-tls-certs" pod "b80e12f0-24c8-4b17-8201-1d6d5b259771" (UID: "b80e12f0-24c8-4b17-8201-1d6d5b259771"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:37:51.762100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.762075 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80e12f0-24c8-4b17-8201-1d6d5b259771-kube-api-access-lqdts" pod "b80e12f0-24c8-4b17-8201-1d6d5b259771" (UID: "b80e12f0-24c8-4b17-8201-1d6d5b259771"). InnerVolumeSpecName "kube-api-access-lqdts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:51.762100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.762085 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-dshm" pod "b80e12f0-24c8-4b17-8201-1d6d5b259771" (UID: "b80e12f0-24c8-4b17-8201-1d6d5b259771"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:37:51.861721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.861683 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80e12f0-24c8-4b17-8201-1d6d5b259771-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:51.861721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.861720 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80e12f0-24c8-4b17-8201-1d6d5b259771-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:51.862293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.861737 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqdts\" (UniqueName: \"kubernetes.io/projected/b80e12f0-24c8-4b17-8201-1d6d5b259771-kube-api-access-lqdts\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:51.901691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.901653 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl"] Oct 31 15:37:51.942650 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:37:51.942624 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2d170c_c6f9_48d5_a496_8f2c9b6f0504.slice/crio-450ef1ac1ec84d714aaec02a7416388b4ffd05362dfb9cb84dbb6ab11fac6acd WatchSource:0}: Error finding container 450ef1ac1ec84d714aaec02a7416388b4ffd05362dfb9cb84dbb6ab11fac6acd: Status 404 returned error can't find the container with id 450ef1ac1ec84d714aaec02a7416388b4ffd05362dfb9cb84dbb6ab11fac6acd Oct 31 15:37:51.950853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:51.950829 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc"] Oct 31 15:37:51.954300 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:37:51.954277 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c77f43_fb3e_4376_9454_a8cbede3f463.slice/crio-6da2d421de51d1e8cf983716e6ef488f86c5270b18e77921306d6d3e6b07702c WatchSource:0}: Error finding container 6da2d421de51d1e8cf983716e6ef488f86c5270b18e77921306d6d3e6b07702c: Status 404 returned error can't find the container with id 6da2d421de51d1e8cf983716e6ef488f86c5270b18e77921306d6d3e6b07702c Oct 31 15:37:52.007519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.007490 4865 generic.go:356] "Generic (PLEG): container finished" podID="b80e12f0-24c8-4b17-8201-1d6d5b259771" containerID="d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4" exitCode=0 Oct 31 15:37:52.007642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.007525 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" event={"ID":"b80e12f0-24c8-4b17-8201-1d6d5b259771","Type":"ContainerDied","Data":"d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4"} Oct 31 15:37:52.007642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.007552 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" Oct 31 15:37:52.007642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.007583 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" event={"ID":"b80e12f0-24c8-4b17-8201-1d6d5b259771","Type":"ContainerDied","Data":"83a21427ec437931e9078ec7e220eb3539f1c22cf784b49e394f61ba33a12d4b"} Oct 31 15:37:52.007642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.007604 4865 scope.go:122] "RemoveContainer" containerID="d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4" Oct 31 15:37:52.009672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.009647 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" event={"ID":"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504","Type":"ContainerStarted","Data":"450ef1ac1ec84d714aaec02a7416388b4ffd05362dfb9cb84dbb6ab11fac6acd"} Oct 31 15:37:52.011692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.011667 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" event={"ID":"48c77f43-fb3e-4376-9454-a8cbede3f463","Type":"ContainerStarted","Data":"6da2d421de51d1e8cf983716e6ef488f86c5270b18e77921306d6d3e6b07702c"} Oct 31 15:37:52.019015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.018997 4865 scope.go:122] "RemoveContainer" containerID="d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4" Oct 31 15:37:52.019302 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:37:52.019278 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4\": container with ID starting with d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4 not found: ID does not exist" containerID="d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4" Oct 31 15:37:52.019412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.019309 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4"} err="failed to get container status \"d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4\": rpc error: code = NotFound desc = could not find container \"d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4\": container with ID starting with d90960eaacd204eec2f344d5a111eab0e6363bbc800d5f64ec6ba379e85764d4 not found: ID does not exist" Oct 31 15:37:52.032374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.032351 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:52.088534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.088498 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj"] Oct 31 15:37:52.092568 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:37:52.092539 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9843c1_ac33_4300_a67c_368c336625d3.slice/crio-aad5f72687e7de89fd0f5202846c605915fe89e2d44a8a092bfe3eeb6b9934bb WatchSource:0}: Error finding container aad5f72687e7de89fd0f5202846c605915fe89e2d44a8a092bfe3eeb6b9934bb: Status 404 returned error can't find the container with id aad5f72687e7de89fd0f5202846c605915fe89e2d44a8a092bfe3eeb6b9934bb Oct 31 15:37:52.175308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.175281 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6"] Oct 31 15:37:52.178739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.178715 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6"] Oct 31 15:37:52.418695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.418665 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt"] Oct 31 15:37:52.419369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.419355 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b80e12f0-24c8-4b17-8201-1d6d5b259771" containerName="main" Oct 31 15:37:52.419428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.419371 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80e12f0-24c8-4b17-8201-1d6d5b259771" containerName="main" Oct 31 15:37:52.419530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.419520 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80e12f0-24c8-4b17-8201-1d6d5b259771" containerName="main" Oct 31 15:37:52.423209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.423193 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.468761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.468730 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrl9\" (UniqueName: \"kubernetes.io/projected/762367c9-5274-40e0-ac0d-32a7da37af9d-kube-api-access-cbrl9\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.468914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.468783 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.468963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.468938 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.479078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.479050 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt"] Oct 31 15:37:52.498665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.498635 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80e12f0-24c8-4b17-8201-1d6d5b259771" path="/var/lib/kubelet/pods/b80e12f0-24c8-4b17-8201-1d6d5b259771/volumes" Oct 31 15:37:52.570628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.570586 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.570863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.570698 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.570972 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.570915 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrl9\" (UniqueName: \"kubernetes.io/projected/762367c9-5274-40e0-ac0d-32a7da37af9d-kube-api-access-cbrl9\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.571516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.571492 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.573634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.573600 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.633084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.633050 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrl9\" (UniqueName: \"kubernetes.io/projected/762367c9-5274-40e0-ac0d-32a7da37af9d-kube-api-access-cbrl9\") pod \"maas-api-revoke-sub-cb478c9f-hzjjt\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:52.676076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.676030 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-55b3fb/auth-test-model-55b3fb-kserve-7d7dffcb89-dh6n6" podUID="b80e12f0-24c8-4b17-8201-1d6d5b259771" containerName="main" probeResult="failure" output="Get \"https://10.132.1.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:37:52.703131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.703097 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:37:52.737242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:52.737152 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:53.020307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.020210 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" event={"ID":"48c77f43-fb3e-4376-9454-a8cbede3f463","Type":"ContainerStarted","Data":"7bc1f65312a3648c99242d954a9a4474dfbb6479e05a076147b11620a3773fe2"} Oct 31 15:37:53.020755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.020314 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:37:53.021902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.021875 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" event={"ID":"fc9843c1-ac33-4300-a67c-368c336625d3","Type":"ContainerStarted","Data":"15542dd8096f94285494ad12d5d9aa403e36af772b42d5fc81e8125c889ba34c"} Oct 31 15:37:53.022023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.021922 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" event={"ID":"fc9843c1-ac33-4300-a67c-368c336625d3","Type":"ContainerStarted","Data":"aad5f72687e7de89fd0f5202846c605915fe89e2d44a8a092bfe3eeb6b9934bb"} Oct 31 15:37:53.022023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.021958 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:37:53.024327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.024298 4865 generic.go:356] "Generic (PLEG): container finished" podID="1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" containerID="19c72823044b9b6023295d8befb48a86a011765aeb78487698b203b28c3dfb6b" exitCode=0 Oct 31 15:37:53.024419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.024366 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" event={"ID":"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504","Type":"ContainerDied","Data":"19c72823044b9b6023295d8befb48a86a011765aeb78487698b203b28c3dfb6b"} Oct 31 15:37:53.123164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.123126 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt"] Oct 31 15:37:53.124033 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:37:53.124005 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762367c9_5274_40e0_ac0d_32a7da37af9d.slice/crio-8a0ba23c397d89741d71210f8a3f53560c456431368830673b97216821e07138 WatchSource:0}: Error finding container 8a0ba23c397d89741d71210f8a3f53560c456431368830673b97216821e07138: Status 404 returned error can't find the container with id 8a0ba23c397d89741d71210f8a3f53560c456431368830673b97216821e07138 Oct 31 15:37:53.141598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.141531 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" podStartSLOduration=2.141508365 podStartE2EDuration="2.141508365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:53.136069522 +0000 UTC m=+3118.755620943" watchObservedRunningTime="2026-10-31 15:37:53.141508365 +0000 UTC m=+3118.761059777" Oct 31 15:37:53.372516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:53.372440 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" podStartSLOduration=2.372426101 podStartE2EDuration="2.372426101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:53.368495984 +0000 UTC m=+3118.988047387" watchObservedRunningTime="2026-10-31 15:37:53.372426101 +0000 UTC m=+3118.991977557" Oct 31 15:37:54.034212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.034170 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" event={"ID":"762367c9-5274-40e0-ac0d-32a7da37af9d","Type":"ContainerStarted","Data":"1967680107eb16024aeaee65f2196a69273f6cb459e19ca79dd10da8e7c7bdc2"} Oct 31 15:37:54.035218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.035189 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" event={"ID":"762367c9-5274-40e0-ac0d-32a7da37af9d","Type":"ContainerStarted","Data":"8a0ba23c397d89741d71210f8a3f53560c456431368830673b97216821e07138"} Oct 31 15:37:54.155128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.155061 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" podStartSLOduration=2.15504079 podStartE2EDuration="2.15504079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:54.150310232 +0000 UTC m=+3119.769861647" watchObservedRunningTime="2026-10-31 15:37:54.15504079 +0000 UTC m=+3119.774592203" Oct 31 15:37:54.198310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.198283 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:54.294855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.294751 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-openshift-ca\") pod \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " Oct 31 15:37:54.294855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.294803 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-kube-api-access-94q7b\" (UniqueName: \"kubernetes.io/projected/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-kube-api-access-94q7b\") pod \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " Oct 31 15:37:54.295119 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.294927 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-xks-ca\") pod \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\" (UID: \"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504\") " Oct 31 15:37:54.295190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.295117 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-openshift-ca" pod "1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" (UID: "1d2d170c-c6f9-48d5-a496-8f2c9b6f0504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:37:54.295427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.295407 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:54.297334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.297310 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-kube-api-access-94q7b" pod "1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" (UID: "1d2d170c-c6f9-48d5-a496-8f2c9b6f0504"). InnerVolumeSpecName "kube-api-access-94q7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:54.297530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.297504 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-xks-ca" pod "1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" (UID: "1d2d170c-c6f9-48d5-a496-8f2c9b6f0504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:37:54.397127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.397084 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:54.397127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:54.397124 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94q7b\" (UniqueName: \"kubernetes.io/projected/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504-kube-api-access-94q7b\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:55.041778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:55.041741 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" Oct 31 15:37:55.041778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:55.041771 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl" event={"ID":"1d2d170c-c6f9-48d5-a496-8f2c9b6f0504","Type":"ContainerDied","Data":"450ef1ac1ec84d714aaec02a7416388b4ffd05362dfb9cb84dbb6ab11fac6acd"} Oct 31 15:37:55.042297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:55.041822 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="450ef1ac1ec84d714aaec02a7416388b4ffd05362dfb9cb84dbb6ab11fac6acd" Oct 31 15:37:55.043528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:55.043504 4865 generic.go:356] "Generic (PLEG): container finished" podID="762367c9-5274-40e0-ac0d-32a7da37af9d" containerID="1967680107eb16024aeaee65f2196a69273f6cb459e19ca79dd10da8e7c7bdc2" exitCode=0 Oct 31 15:37:55.043661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:55.043582 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" event={"ID":"762367c9-5274-40e0-ac0d-32a7da37af9d","Type":"ContainerDied","Data":"1967680107eb16024aeaee65f2196a69273f6cb459e19ca79dd10da8e7c7bdc2"} Oct 31 15:37:56.203414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.203386 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:56.324169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.324087 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762367c9-5274-40e0-ac0d-32a7da37af9d-kube-api-access-cbrl9\" (UniqueName: \"kubernetes.io/projected/762367c9-5274-40e0-ac0d-32a7da37af9d-kube-api-access-cbrl9\") pod \"762367c9-5274-40e0-ac0d-32a7da37af9d\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " Oct 31 15:37:56.324331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.324214 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-openshift-ca\") pod \"762367c9-5274-40e0-ac0d-32a7da37af9d\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " Oct 31 15:37:56.324437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.324420 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-xks-ca\") pod \"762367c9-5274-40e0-ac0d-32a7da37af9d\" (UID: \"762367c9-5274-40e0-ac0d-32a7da37af9d\") " Oct 31 15:37:56.326617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.326571 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-openshift-ca" pod "762367c9-5274-40e0-ac0d-32a7da37af9d" (UID: "762367c9-5274-40e0-ac0d-32a7da37af9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:37:56.327809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.327781 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-xks-ca" pod "762367c9-5274-40e0-ac0d-32a7da37af9d" (UID: "762367c9-5274-40e0-ac0d-32a7da37af9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:37:56.329741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.329713 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762367c9-5274-40e0-ac0d-32a7da37af9d-kube-api-access-cbrl9" pod "762367c9-5274-40e0-ac0d-32a7da37af9d" (UID: "762367c9-5274-40e0-ac0d-32a7da37af9d"). InnerVolumeSpecName "kube-api-access-cbrl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:37:56.426590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.426553 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:56.426590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.426586 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762367c9-5274-40e0-ac0d-32a7da37af9d-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:56.426788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.426600 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrl9\" (UniqueName: \"kubernetes.io/projected/762367c9-5274-40e0-ac0d-32a7da37af9d-kube-api-access-cbrl9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:37:56.789200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:56.789158 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:37:57.065455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:57.065372 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" Oct 31 15:37:57.065658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:57.065369 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt" event={"ID":"762367c9-5274-40e0-ac0d-32a7da37af9d","Type":"ContainerDied","Data":"8a0ba23c397d89741d71210f8a3f53560c456431368830673b97216821e07138"} Oct 31 15:37:57.065658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:37:57.065533 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0ba23c397d89741d71210f8a3f53560c456431368830673b97216821e07138" Oct 31 15:38:01.460930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:01.460887 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:02.757361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:02.757330 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:04.037769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:04.037738 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:38:04.038162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:04.037807 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:38:04.198885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:04.198855 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:06.104661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:06.104629 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:10.795002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:10.794969 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:11.909832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:11.909796 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w"] Oct 31 15:38:11.927827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:11.927745 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:15.492145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:15.492109 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:17.492058 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:38:17.492014 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:38:20.249654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:20.249614 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:24.915163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:24.915125 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:26.931927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:26.931869 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" podUID="f94a13b8-0e82-44da-80fd-655838216548" containerName="main" containerID="cri-o://371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4" gracePeriod=45 Oct 31 15:38:27.198456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.198399 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:38:27.268050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268015 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-model-cache\") pod \"f94a13b8-0e82-44da-80fd-655838216548\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " Oct 31 15:38:27.268194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268137 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94a13b8-0e82-44da-80fd-655838216548-kube-api-access-w2xx6\" (UniqueName: \"kubernetes.io/projected/f94a13b8-0e82-44da-80fd-655838216548-kube-api-access-w2xx6\") pod \"f94a13b8-0e82-44da-80fd-655838216548\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " Oct 31 15:38:27.268194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268187 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-dshm\") pod \"f94a13b8-0e82-44da-80fd-655838216548\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " Oct 31 15:38:27.268320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268215 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-tmp-dir\") pod \"f94a13b8-0e82-44da-80fd-655838216548\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " Oct 31 15:38:27.268320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268267 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-model-cache" pod "f94a13b8-0e82-44da-80fd-655838216548" (UID: "f94a13b8-0e82-44da-80fd-655838216548"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:27.268449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268329 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f94a13b8-0e82-44da-80fd-655838216548-tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a13b8-0e82-44da-80fd-655838216548-tls-certs\") pod \"f94a13b8-0e82-44da-80fd-655838216548\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " Oct 31 15:38:27.268449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268443 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-tmp-dir" pod "f94a13b8-0e82-44da-80fd-655838216548" (UID: "f94a13b8-0e82-44da-80fd-655838216548"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:27.268671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268459 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-home\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-home\") pod \"f94a13b8-0e82-44da-80fd-655838216548\" (UID: \"f94a13b8-0e82-44da-80fd-655838216548\") " Oct 31 15:38:27.268671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268638 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-home" pod "f94a13b8-0e82-44da-80fd-655838216548" (UID: "f94a13b8-0e82-44da-80fd-655838216548"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:27.268984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268932 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:27.268984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268950 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:27.268984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.268959 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:27.270670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.270646 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94a13b8-0e82-44da-80fd-655838216548-tls-certs" pod "f94a13b8-0e82-44da-80fd-655838216548" (UID: "f94a13b8-0e82-44da-80fd-655838216548"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:38:27.270776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.270698 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-dshm" pod "f94a13b8-0e82-44da-80fd-655838216548" (UID: "f94a13b8-0e82-44da-80fd-655838216548"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:27.270776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.270720 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94a13b8-0e82-44da-80fd-655838216548-kube-api-access-w2xx6" pod "f94a13b8-0e82-44da-80fd-655838216548" (UID: "f94a13b8-0e82-44da-80fd-655838216548"). InnerVolumeSpecName "kube-api-access-w2xx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:38:27.354571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.354537 4865 generic.go:356] "Generic (PLEG): container finished" podID="f94a13b8-0e82-44da-80fd-655838216548" containerID="371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4" exitCode=0 Oct 31 15:38:27.354736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.354584 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" Oct 31 15:38:27.354736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.354621 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" event={"ID":"f94a13b8-0e82-44da-80fd-655838216548","Type":"ContainerDied","Data":"371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4"} Oct 31 15:38:27.354736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.354665 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w" event={"ID":"f94a13b8-0e82-44da-80fd-655838216548","Type":"ContainerDied","Data":"07eca2301b7a8447b9727c305c0f6b16c92141ac0c5b5778fa7d740d3642a016"} Oct 31 15:38:27.354736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.354682 4865 scope.go:122] "RemoveContainer" containerID="371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4" Oct 31 15:38:27.366159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.366140 4865 scope.go:122] "RemoveContainer" containerID="371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4" Oct 31 15:38:27.366456 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:38:27.366430 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4\": container with ID starting with 371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4 not found: ID does not exist" containerID="371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4" Oct 31 15:38:27.366555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.366464 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4"} err="failed to get container status \"371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4\": rpc error: code = NotFound desc = could not find container \"371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4\": container with ID starting with 371e6621756f549c69e341c4a789ee86f89bd4992cc86de24f93b2236b5b64e4 not found: ID does not exist" Oct 31 15:38:27.370280 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.370263 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xx6\" (UniqueName: \"kubernetes.io/projected/f94a13b8-0e82-44da-80fd-655838216548-kube-api-access-w2xx6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:27.370361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.370282 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94a13b8-0e82-44da-80fd-655838216548-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:27.370361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.370291 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94a13b8-0e82-44da-80fd-655838216548-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:27.522115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.522091 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w"] Oct 31 15:38:27.525880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:27.525850 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/auth-test-model-e72453-kserve-77c98db4fc-dtv7w"] Oct 31 15:38:28.498151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:28.498121 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94a13b8-0e82-44da-80fd-655838216548" path="/var/lib/kubelet/pods/f94a13b8-0e82-44da-80fd-655838216548/volumes" Oct 31 15:38:28.574379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:28.574345 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:29.805152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:29.805117 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:34.431729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:34.431692 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:37.584183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:37.584152 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64"] Oct 31 15:38:37.585833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:37.585802 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:39.106337 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:39.106293 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:43.798674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:43.798630 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:44.078166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.078092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:44.566263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.566227 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj"] Oct 31 15:38:44.566962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.566944 4865 cpu_manager.go:414] "containerMap: removing container" podUID="762367c9-5274-40e0-ac0d-32a7da37af9d" containerName="revoke-keys" Oct 31 15:38:44.567036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.566966 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762367c9-5274-40e0-ac0d-32a7da37af9d" containerName="revoke-keys" Oct 31 15:38:44.567036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.566994 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f94a13b8-0e82-44da-80fd-655838216548" containerName="main" Oct 31 15:38:44.567036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.567002 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94a13b8-0e82-44da-80fd-655838216548" containerName="main" Oct 31 15:38:44.567178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.567041 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" containerName="revoke-keys" Oct 31 15:38:44.567178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.567049 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" containerName="revoke-keys" Oct 31 15:38:44.567275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.567250 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" containerName="revoke-keys" Oct 31 15:38:44.567326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.567281 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="762367c9-5274-40e0-ac0d-32a7da37af9d" containerName="revoke-keys" Oct 31 15:38:44.567326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.567294 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94a13b8-0e82-44da-80fd-655838216548" containerName="main" Oct 31 15:38:44.572720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.572704 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.642921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.642884 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:44.655394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.655364 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-4d1877-openshift-default-dockercfg-vxjqv\"" Oct 31 15:38:44.680212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680182 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-podinfo\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680224 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-token\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680255 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-data\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680303 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97kc\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-kube-api-access-r97kc\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680332 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-certs\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680372 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-credential-socket\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680495 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-envoy\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680559 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-socket\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.680703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.680686 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e3f80f-a50e-4b21-87da-4e069cf41144-istiod-ca-cert\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.684811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.684785 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj"] Oct 31 15:38:44.782269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782221 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e3f80f-a50e-4b21-87da-4e069cf41144-istiod-ca-cert\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782304 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-podinfo\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782334 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-token\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782351 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-data\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782387 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97kc\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-kube-api-access-r97kc\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782410 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-certs\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782437 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-credential-socket\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782534 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-envoy\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782575 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-socket\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.782998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.782964 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-socket\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.783420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.783398 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e3f80f-a50e-4b21-87da-4e069cf41144-istiod-ca-cert\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.784005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.783983 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-data\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.784005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.783998 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-credential-socket\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.784239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.784218 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-certs\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.786094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.786066 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-podinfo\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.786342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.786322 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-envoy\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.843840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.843770 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-token\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.844175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.843899 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97kc\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-kube-api-access-r97kc\") pod \"e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.887663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.887162 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:44.991747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:44.991700 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:45.318856 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:38:45.318828 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e3f80f_a50e_4b21_87da_4e069cf41144.slice/crio-62aad49a838a464375f3ae7cc71580250657a3445ed0a9e936fe05fa44cf2ead WatchSource:0}: Error finding container 62aad49a838a464375f3ae7cc71580250657a3445ed0a9e936fe05fa44cf2ead: Status 404 returned error can't find the container with id 62aad49a838a464375f3ae7cc71580250657a3445ed0a9e936fe05fa44cf2ead Oct 31 15:38:45.319221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.319198 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj"] Oct 31 15:38:45.321346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.321312 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:38:45.321451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.321374 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:38:45.321451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.321404 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:38:45.482172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.482134 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" event={"ID":"c9e3f80f-a50e-4b21-87da-4e069cf41144","Type":"ContainerStarted","Data":"9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843"} Oct 31 15:38:45.482391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.482191 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" event={"ID":"c9e3f80f-a50e-4b21-87da-4e069cf41144","Type":"ContainerStarted","Data":"62aad49a838a464375f3ae7cc71580250657a3445ed0a9e936fe05fa44cf2ead"} Oct 31 15:38:45.600003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.599897 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" podStartSLOduration=1.599877341 podStartE2EDuration="1.599877341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:38:45.598915068 +0000 UTC m=+3171.218466483" watchObservedRunningTime="2026-10-31 15:38:45.599877341 +0000 UTC m=+3171.219428753" Oct 31 15:38:45.887921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:45.887852 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:46.894153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:46.894123 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:47.491420 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:38:47.491389 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:38:47.499439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:47.499415 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:47.500357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:47.500335 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:38:49.288706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:49.288675 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:49.411418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:49.411387 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:51.541449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:51.541417 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:51.695665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:51.695630 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:51.938806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:51.938729 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l"] Oct 31 15:38:51.943349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:51.943334 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:51.987687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:51.987663 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l"] Oct 31 15:38:51.996635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:51.996606 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-4d1877-dockercfg-bl6zz\"" Oct 31 15:38:51.996849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:51.996824 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-4d1877\"" Oct 31 15:38:52.004832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.004810 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:38:52.077818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.077784 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91685308-585b-4a27-9636-bb675ddda875-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:52.078127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.078107 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrl7t\" (UniqueName: \"kubernetes.io/projected/91685308-585b-4a27-9636-bb675ddda875-kube-api-access-mrl7t\") pod \"payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:52.101802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.101770 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2"] Oct 31 15:38:52.106597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.106575 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.163412 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.163383 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2"] Oct 31 15:38:52.179513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.179484 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91685308-585b-4a27-9636-bb675ddda875-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:52.179676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.179592 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrl7t\" (UniqueName: \"kubernetes.io/projected/91685308-585b-4a27-9636-bb675ddda875-kube-api-access-mrl7t\") pod \"payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:52.180352 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.180327 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91685308-585b-4a27-9636-bb675ddda875-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:52.237704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.237677 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrl7t\" (UniqueName: \"kubernetes.io/projected/91685308-585b-4a27-9636-bb675ddda875-kube-api-access-mrl7t\") pod \"payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:52.258574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.258552 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:52.280688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.280663 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47t9\" (UniqueName: \"kubernetes.io/projected/c97da477-d93f-45a3-b56c-0ce7be5cc00a-kube-api-access-k47t9\") pod \"payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.280814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.280736 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97da477-d93f-45a3-b56c-0ce7be5cc00a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.386664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.386592 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47t9\" (UniqueName: \"kubernetes.io/projected/c97da477-d93f-45a3-b56c-0ce7be5cc00a-kube-api-access-k47t9\") pod \"payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.386869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.386717 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97da477-d93f-45a3-b56c-0ce7be5cc00a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.387861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.387829 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97da477-d93f-45a3-b56c-0ce7be5cc00a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.452930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.449714 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47t9\" (UniqueName: \"kubernetes.io/projected/c97da477-d93f-45a3-b56c-0ce7be5cc00a-kube-api-access-k47t9\") pod \"payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.605648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.605562 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" podUID="5f595fb7-0f07-4e54-97ad-e61c8d71a874" containerName="main" containerID="cri-o://d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a" gracePeriod=45 Oct 31 15:38:52.647134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.647099 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l"] Oct 31 15:38:52.662493 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:38:52.662451 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91685308_585b_4a27_9636_bb675ddda875.slice/crio-91b701aa2cf83464bc977870abc2c9d4de0d60d3bb04f49fff4512f8ff0cdbe9 WatchSource:0}: Error finding container 91b701aa2cf83464bc977870abc2c9d4de0d60d3bb04f49fff4512f8ff0cdbe9: Status 404 returned error can't find the container with id 91b701aa2cf83464bc977870abc2c9d4de0d60d3bb04f49fff4512f8ff0cdbe9 Oct 31 15:38:52.724289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.724249 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:52.930944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:52.930919 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:38:53.097185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097144 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-home\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-home\") pod \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " Oct 31 15:38:53.097369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097206 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-dshm\") pod \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " Oct 31 15:38:53.097369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097338 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f595fb7-0f07-4e54-97ad-e61c8d71a874-kube-api-access-9wtvc\" (UniqueName: \"kubernetes.io/projected/5f595fb7-0f07-4e54-97ad-e61c8d71a874-kube-api-access-9wtvc\") pod \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " Oct 31 15:38:53.097534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097392 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tls-certs\") pod \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " Oct 31 15:38:53.097534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097437 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-home" pod "5f595fb7-0f07-4e54-97ad-e61c8d71a874" (UID: "5f595fb7-0f07-4e54-97ad-e61c8d71a874"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:53.097650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097593 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-model-cache\") pod \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " Oct 31 15:38:53.097719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097661 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tmp-dir\") pod \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\" (UID: \"5f595fb7-0f07-4e54-97ad-e61c8d71a874\") " Oct 31 15:38:53.097993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.097966 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-model-cache" pod "5f595fb7-0f07-4e54-97ad-e61c8d71a874" (UID: "5f595fb7-0f07-4e54-97ad-e61c8d71a874"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:53.098106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.098052 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tmp-dir" pod "5f595fb7-0f07-4e54-97ad-e61c8d71a874" (UID: "5f595fb7-0f07-4e54-97ad-e61c8d71a874"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:53.098397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.098379 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:53.098525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.098401 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:53.098525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.098417 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:53.100445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.100420 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tls-certs" pod "5f595fb7-0f07-4e54-97ad-e61c8d71a874" (UID: "5f595fb7-0f07-4e54-97ad-e61c8d71a874"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:38:53.100569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.100520 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-dshm" pod "5f595fb7-0f07-4e54-97ad-e61c8d71a874" (UID: "5f595fb7-0f07-4e54-97ad-e61c8d71a874"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:38:53.100649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.100627 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f595fb7-0f07-4e54-97ad-e61c8d71a874-kube-api-access-9wtvc" pod "5f595fb7-0f07-4e54-97ad-e61c8d71a874" (UID: "5f595fb7-0f07-4e54-97ad-e61c8d71a874"). InnerVolumeSpecName "kube-api-access-9wtvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:38:53.117636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.117604 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2"] Oct 31 15:38:53.119137 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:38:53.119116 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97da477_d93f_45a3_b56c_0ce7be5cc00a.slice/crio-c220a2a119109c90c646dccb34880cf5d3efd1f4ff08708f11eb3bfee739dc3a WatchSource:0}: Error finding container c220a2a119109c90c646dccb34880cf5d3efd1f4ff08708f11eb3bfee739dc3a: Status 404 returned error can't find the container with id c220a2a119109c90c646dccb34880cf5d3efd1f4ff08708f11eb3bfee739dc3a Oct 31 15:38:53.199303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.199280 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wtvc\" (UniqueName: \"kubernetes.io/projected/5f595fb7-0f07-4e54-97ad-e61c8d71a874-kube-api-access-9wtvc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:53.199406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.199307 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f595fb7-0f07-4e54-97ad-e61c8d71a874-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:53.199406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.199316 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f595fb7-0f07-4e54-97ad-e61c8d71a874-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:38:53.544756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.544725 4865 generic.go:356] "Generic (PLEG): container finished" podID="5f595fb7-0f07-4e54-97ad-e61c8d71a874" containerID="d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a" exitCode=0 Oct 31 15:38:53.544969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.544772 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" Oct 31 15:38:53.544969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.544813 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" event={"ID":"5f595fb7-0f07-4e54-97ad-e61c8d71a874","Type":"ContainerDied","Data":"d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a"} Oct 31 15:38:53.544969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.544858 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64" event={"ID":"5f595fb7-0f07-4e54-97ad-e61c8d71a874","Type":"ContainerDied","Data":"e967b068cd9e1bf011300fa3ff44783dd77f9780b84aa167b8c1ea6b8b0cc1fb"} Oct 31 15:38:53.544969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.544877 4865 scope.go:122] "RemoveContainer" containerID="d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a" Oct 31 15:38:53.547078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.547039 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" event={"ID":"91685308-585b-4a27-9636-bb675ddda875","Type":"ContainerStarted","Data":"87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d"} Oct 31 15:38:53.547197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.547105 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" event={"ID":"91685308-585b-4a27-9636-bb675ddda875","Type":"ContainerStarted","Data":"91b701aa2cf83464bc977870abc2c9d4de0d60d3bb04f49fff4512f8ff0cdbe9"} Oct 31 15:38:53.547197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.547139 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:38:53.549144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.549122 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" event={"ID":"c97da477-d93f-45a3-b56c-0ce7be5cc00a","Type":"ContainerStarted","Data":"2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615"} Oct 31 15:38:53.549278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.549177 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" event={"ID":"c97da477-d93f-45a3-b56c-0ce7be5cc00a","Type":"ContainerStarted","Data":"c220a2a119109c90c646dccb34880cf5d3efd1f4ff08708f11eb3bfee739dc3a"} Oct 31 15:38:53.549278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.549198 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:38:53.556312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.556296 4865 scope.go:122] "RemoveContainer" containerID="d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a" Oct 31 15:38:53.556563 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:38:53.556541 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a\": container with ID starting with d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a not found: ID does not exist" containerID="d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a" Oct 31 15:38:53.556627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.556571 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a"} err="failed to get container status \"d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a\": rpc error: code = NotFound desc = could not find container \"d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a\": container with ID starting with d64205eb60a2e81aedaa2c316a5b287400002a2ef998b7462f5a5e7f731e800a not found: ID does not exist" Oct 31 15:38:53.607179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.607153 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:53.713252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.713224 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64"] Oct 31 15:38:53.716929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.716906 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-21111f/auth-test-model-21111f-kserve-779968577c-swb64"] Oct 31 15:38:53.829850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.829725 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" podStartSLOduration=1.829712503 podStartE2EDuration="1.829712503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:38:53.826706143 +0000 UTC m=+3179.446257554" watchObservedRunningTime="2026-10-31 15:38:53.829712503 +0000 UTC m=+3179.449263913" Oct 31 15:38:53.943348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:53.943282 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" podStartSLOduration=2.943262608 podStartE2EDuration="2.943262608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:38:53.942227708 +0000 UTC m=+3179.561779121" watchObservedRunningTime="2026-10-31 15:38:53.943262608 +0000 UTC m=+3179.562814019" Oct 31 15:38:54.500699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:54.500653 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f595fb7-0f07-4e54-97ad-e61c8d71a874" path="/var/lib/kubelet/pods/5f595fb7-0f07-4e54-97ad-e61c8d71a874/volumes" Oct 31 15:38:58.143568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:58.143536 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:58.314979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:58.314942 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:38:58.514357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:38:58.514327 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:02.648862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:02.648828 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:02.999022 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:02.998994 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:03.119530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:03.119495 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:04.530206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:04.530174 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:04.560257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:04.560233 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:39:04.560519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:04.560503 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:39:07.344892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.344815 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:07.714439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.714400 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:07.872762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.872732 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645"] Oct 31 15:39:07.873435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.873421 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5f595fb7-0f07-4e54-97ad-e61c8d71a874" containerName="main" Oct 31 15:39:07.873576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.873437 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f595fb7-0f07-4e54-97ad-e61c8d71a874" containerName="main" Oct 31 15:39:07.873576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.873552 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f595fb7-0f07-4e54-97ad-e61c8d71a874" containerName="main" Oct 31 15:39:07.878454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.878434 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:07.968693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.966290 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645"] Oct 31 15:39:07.980522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.980486 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:07.984441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:07.984406 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c59395-openshift-default-dockercfg-v2z9x\"" Oct 31 15:39:08.084303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084257 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-podinfo\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084323 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-certs\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084425 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vk5\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-kube-api-access-q9vk5\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084541 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-credential-socket\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084597 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9036291-7198-4aca-8a3d-84ef77b0705f-istiod-ca-cert\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084686 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-token\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084709 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-data\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084760 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-envoy\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.084860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.084830 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-socket\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.174131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.174097 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:08.185904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.185879 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-token\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.185917 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-data\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.185986 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-envoy\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186022 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-socket\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186185 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-podinfo\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186256 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186229 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-certs\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186268 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vk5\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-kube-api-access-q9vk5\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186351 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186336 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-credential-socket\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186361 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9036291-7198-4aca-8a3d-84ef77b0705f-istiod-ca-cert\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186429 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-data\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186445 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-socket\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186863 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-certs\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.186464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.186906 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-credential-socket\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.187401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.187381 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9036291-7198-4aca-8a3d-84ef77b0705f-istiod-ca-cert\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.189053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.189032 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-envoy\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.189296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.189277 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-podinfo\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.244017 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.243952 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-token\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.244141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.244071 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vk5\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-kube-api-access-q9vk5\") pod \"e2e-extmod-a-c59395-openshift-default-547997b685-sd645\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.493004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.492971 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:08.931616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.931580 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645"] Oct 31 15:39:08.935324 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:39:08.935294 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9036291_7198_4aca_8a3d_84ef77b0705f.slice/crio-72e81e7ae02dc637e67a04606c8a9a8a2e324effeb25eac1459c52bbe2408466 WatchSource:0}: Error finding container 72e81e7ae02dc637e67a04606c8a9a8a2e324effeb25eac1459c52bbe2408466: Status 404 returned error can't find the container with id 72e81e7ae02dc637e67a04606c8a9a8a2e324effeb25eac1459c52bbe2408466 Oct 31 15:39:08.938369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.938314 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:39:08.938512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.938421 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:39:08.938512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.938488 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:39:08.965687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:08.965657 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:09.672743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:09.672707 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" event={"ID":"c9036291-7198-4aca-8a3d-84ef77b0705f","Type":"ContainerStarted","Data":"781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f"} Oct 31 15:39:09.673255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:09.672764 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" event={"ID":"c9036291-7198-4aca-8a3d-84ef77b0705f","Type":"ContainerStarted","Data":"72e81e7ae02dc637e67a04606c8a9a8a2e324effeb25eac1459c52bbe2408466"} Oct 31 15:39:09.684342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:09.684314 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:09.830070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:09.830018 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podStartSLOduration=2.8300041609999997 podStartE2EDuration="2.830004161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:39:09.809308167 +0000 UTC m=+3195.428859577" watchObservedRunningTime="2026-10-31 15:39:09.830004161 +0000 UTC m=+3195.449555571" Oct 31 15:39:10.497898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:10.497869 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:10.499841 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:10.499821 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:10.679966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:10.679855 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:10.680856 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:10.680834 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:39:12.594258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:12.594220 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:12.794736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:12.794698 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:12.991368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:12.991336 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:17.492054 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:39:17.492017 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:39:17.667883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:17.667836 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:17.824618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:17.824519 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:17.944789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:17.944747 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:18.092755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.092675 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf"] Oct 31 15:39:18.117496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.112234 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.153912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.153878 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf"] Oct 31 15:39:18.203493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.202808 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c59395-dockercfg-5jgf8\"" Oct 31 15:39:18.203493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.202906 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c59395\"" Oct 31 15:39:18.262670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.262633 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgq2\" (UniqueName: \"kubernetes.io/projected/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-kube-api-access-lvgq2\") pod \"payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.263246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.263157 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.273314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.273286 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc"] Oct 31 15:39:18.279265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.279248 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.332454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.332421 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc"] Oct 31 15:39:18.364869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.364792 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgq2\" (UniqueName: \"kubernetes.io/projected/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-kube-api-access-lvgq2\") pod \"payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.365348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.365304 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bhs\" (UniqueName: \"kubernetes.io/projected/67f3fbff-e845-47e8-a338-eb89c17e9ba5-kube-api-access-b2bhs\") pod \"payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.365519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.365498 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.365910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.365892 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f3fbff-e845-47e8-a338-eb89c17e9ba5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.366122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.366090 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.425846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.425817 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgq2\" (UniqueName: \"kubernetes.io/projected/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-kube-api-access-lvgq2\") pod \"payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.434148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.434126 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:18.467491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.467447 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f3fbff-e845-47e8-a338-eb89c17e9ba5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.467668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.467588 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bhs\" (UniqueName: \"kubernetes.io/projected/67f3fbff-e845-47e8-a338-eb89c17e9ba5-kube-api-access-b2bhs\") pod \"payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.468918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.468895 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f3fbff-e845-47e8-a338-eb89c17e9ba5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.529354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.529325 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bhs\" (UniqueName: \"kubernetes.io/projected/67f3fbff-e845-47e8-a338-eb89c17e9ba5-kube-api-access-b2bhs\") pod \"payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.617553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.616721 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:18.824862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:18.824834 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf"] Oct 31 15:39:18.827337 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:39:18.827300 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a7f5a9_9c5b_499a_a6ca_ffaf0524e96e.slice/crio-6233380670f226339520a3b274488527cc63457eba9893ee83c9a0093c2bc3ad WatchSource:0}: Error finding container 6233380670f226339520a3b274488527cc63457eba9893ee83c9a0093c2bc3ad: Status 404 returned error can't find the container with id 6233380670f226339520a3b274488527cc63457eba9893ee83c9a0093c2bc3ad Oct 31 15:39:19.020986 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:39:19.020898 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f3fbff_e845_47e8_a338_eb89c17e9ba5.slice/crio-f2a2275a73990411eb150d2d0ea05e9a4936a2d95304cc8f2cff0d3bc688c08a WatchSource:0}: Error finding container f2a2275a73990411eb150d2d0ea05e9a4936a2d95304cc8f2cff0d3bc688c08a: Status 404 returned error can't find the container with id f2a2275a73990411eb150d2d0ea05e9a4936a2d95304cc8f2cff0d3bc688c08a Oct 31 15:39:19.024651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.024625 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc"] Oct 31 15:39:19.762208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.762168 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" event={"ID":"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e","Type":"ContainerStarted","Data":"719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2"} Oct 31 15:39:19.762694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.762241 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:19.762694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.762272 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" event={"ID":"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e","Type":"ContainerStarted","Data":"6233380670f226339520a3b274488527cc63457eba9893ee83c9a0093c2bc3ad"} Oct 31 15:39:19.764152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.764121 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" event={"ID":"67f3fbff-e845-47e8-a338-eb89c17e9ba5","Type":"ContainerStarted","Data":"3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031"} Oct 31 15:39:19.764290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.764179 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" event={"ID":"67f3fbff-e845-47e8-a338-eb89c17e9ba5","Type":"ContainerStarted","Data":"f2a2275a73990411eb150d2d0ea05e9a4936a2d95304cc8f2cff0d3bc688c08a"} Oct 31 15:39:19.764290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.764209 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:19.880252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:19.880189 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" podStartSLOduration=1.8801743069999999 podStartE2EDuration="1.880174307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:39:19.878211931 +0000 UTC m=+3205.497763342" watchObservedRunningTime="2026-10-31 15:39:19.880174307 +0000 UTC m=+3205.499725717" Oct 31 15:39:20.012317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:20.012183 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" podStartSLOduration=2.012158606 podStartE2EDuration="2.012158606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:39:19.993460527 +0000 UTC m=+3205.613011937" watchObservedRunningTime="2026-10-31 15:39:20.012158606 +0000 UTC m=+3205.631710018" Oct 31 15:39:22.497106 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:22.497073 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:24.803632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:24.803592 4865 generic.go:356] "Generic (PLEG): container finished" podID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerID="5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424" exitCode=0 Oct 31 15:39:24.804175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:24.803666 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" event={"ID":"2c868694-55bc-4cb9-83cb-03e07dfe35ba","Type":"ContainerDied","Data":"5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424"} Oct 31 15:39:24.804175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:24.804068 4865 scope.go:122] "RemoveContainer" containerID="5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424" Oct 31 15:39:24.805690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:24.805662 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:39:24.805765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:24.805742 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:39:24.805803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:24.805775 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:39:25.825848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:25.825810 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" event={"ID":"2c868694-55bc-4cb9-83cb-03e07dfe35ba","Type":"ContainerStarted","Data":"51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8"} Oct 31 15:39:26.618496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:26.617854 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:39:26.619429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:26.619397 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 15:39:26.619615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:26.619454 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 15:39:26.926530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:26.926418 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:39:27.218588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:27.209950 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:27.615552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:27.614889 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 15:39:27.615552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:27.614966 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 15:39:28.618497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:28.618358 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 15:39:28.618497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:28.618426 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 15:39:29.618497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:29.617785 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 15:39:29.618497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:29.617857 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 15:39:30.462437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:30.462406 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb"] Oct 31 15:39:30.468886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:30.468861 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:30.618502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:30.614825 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 15:39:30.618502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:30.614886 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 15:39:30.774842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:30.774769 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:39:30.774842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:30.774838 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:39:31.600396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:31.600368 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:39:31.601298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:31.601277 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:39:31.851052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:31.850972 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:36.490193 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:36.490157 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:41.115868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:41.115837 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:45.136662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.136628 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:39:45.484132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.484084 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" podUID="979de514-5322-4324-9a94-46f79c2e6c65" containerName="main" containerID="cri-o://d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34" gracePeriod=45 Oct 31 15:39:45.764691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.764666 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:39:45.815959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.815924 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:45.834535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834511 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/979de514-5322-4324-9a94-46f79c2e6c65-tls-certs\" (UniqueName: \"kubernetes.io/secret/979de514-5322-4324-9a94-46f79c2e6c65-tls-certs\") pod \"979de514-5322-4324-9a94-46f79c2e6c65\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " Oct 31 15:39:45.834662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834591 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-tmp-dir\") pod \"979de514-5322-4324-9a94-46f79c2e6c65\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " Oct 31 15:39:45.834662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834633 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979de514-5322-4324-9a94-46f79c2e6c65-kube-api-access-m57vx\" (UniqueName: \"kubernetes.io/projected/979de514-5322-4324-9a94-46f79c2e6c65-kube-api-access-m57vx\") pod \"979de514-5322-4324-9a94-46f79c2e6c65\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " Oct 31 15:39:45.834749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834669 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-home\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-home\") pod \"979de514-5322-4324-9a94-46f79c2e6c65\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " Oct 31 15:39:45.834836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834821 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-dshm\") pod \"979de514-5322-4324-9a94-46f79c2e6c65\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " Oct 31 15:39:45.834915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834858 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-tmp-dir" pod "979de514-5322-4324-9a94-46f79c2e6c65" (UID: "979de514-5322-4324-9a94-46f79c2e6c65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:39:45.834915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834885 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-model-cache\") pod \"979de514-5322-4324-9a94-46f79c2e6c65\" (UID: \"979de514-5322-4324-9a94-46f79c2e6c65\") " Oct 31 15:39:45.835043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.834955 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-home" pod "979de514-5322-4324-9a94-46f79c2e6c65" (UID: "979de514-5322-4324-9a94-46f79c2e6c65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:39:45.835101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.835068 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-model-cache" pod "979de514-5322-4324-9a94-46f79c2e6c65" (UID: "979de514-5322-4324-9a94-46f79c2e6c65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:39:45.835778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.835752 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:39:45.835891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.835782 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:39:45.835891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.835800 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:39:45.837152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.837123 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-dshm" pod "979de514-5322-4324-9a94-46f79c2e6c65" (UID: "979de514-5322-4324-9a94-46f79c2e6c65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:39:45.837247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.837154 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979de514-5322-4324-9a94-46f79c2e6c65-tls-certs" pod "979de514-5322-4324-9a94-46f79c2e6c65" (UID: "979de514-5322-4324-9a94-46f79c2e6c65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:39:45.837296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.837240 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979de514-5322-4324-9a94-46f79c2e6c65-kube-api-access-m57vx" pod "979de514-5322-4324-9a94-46f79c2e6c65" (UID: "979de514-5322-4324-9a94-46f79c2e6c65"). InnerVolumeSpecName "kube-api-access-m57vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:39:45.936971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.936934 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m57vx\" (UniqueName: \"kubernetes.io/projected/979de514-5322-4324-9a94-46f79c2e6c65-kube-api-access-m57vx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:39:45.936971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.936966 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979de514-5322-4324-9a94-46f79c2e6c65-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:39:45.936971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.936977 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979de514-5322-4324-9a94-46f79c2e6c65-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:39:45.974391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.974363 4865 generic.go:356] "Generic (PLEG): container finished" podID="979de514-5322-4324-9a94-46f79c2e6c65" containerID="d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34" exitCode=0 Oct 31 15:39:45.974589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.974409 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" Oct 31 15:39:45.974589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.974441 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" event={"ID":"979de514-5322-4324-9a94-46f79c2e6c65","Type":"ContainerDied","Data":"d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34"} Oct 31 15:39:45.974589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.974503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb" event={"ID":"979de514-5322-4324-9a94-46f79c2e6c65","Type":"ContainerDied","Data":"4687bfb81d88a1583fd23831ce5699362ffeaff4787b94751ec4b17509f8fbae"} Oct 31 15:39:45.974589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.974520 4865 scope.go:122] "RemoveContainer" containerID="d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34" Oct 31 15:39:45.985946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.985929 4865 scope.go:122] "RemoveContainer" containerID="d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34" Oct 31 15:39:45.986225 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:39:45.986205 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34\": container with ID starting with d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34 not found: ID does not exist" containerID="d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34" Oct 31 15:39:45.986292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:45.986233 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34"} err="failed to get container status \"d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34\": rpc error: code = NotFound desc = could not find container \"d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34\": container with ID starting with d4a02da897f123823d080068ee4cca9e20b67754e949f35807a01b69b9bbcb34 not found: ID does not exist" Oct 31 15:39:46.149833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:46.149749 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb"] Oct 31 15:39:46.158913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:46.158884 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-46bf36/auth-test-model-46bf36-kserve-7f64655fb5-ktklb"] Oct 31 15:39:46.497090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:46.497060 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979de514-5322-4324-9a94-46f79c2e6c65" path="/var/lib/kubelet/pods/979de514-5322-4324-9a94-46f79c2e6c65/volumes" Oct 31 15:39:47.491265 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:39:47.491224 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:39:50.447030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:50.446988 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:52.462110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:52.462077 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:55.185164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:55.185132 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:57.524325 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:57.524291 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:39:59.716422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:39:59.716386 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:03.436905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:03.436871 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:04.259914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:04.259885 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:04.418355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:04.418319 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:04.544663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:04.544582 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:04.872879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:04.872797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:09.115646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:09.115614 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:10.885312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:10.885279 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:13.276920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:13.276885 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:13.803569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:13.803530 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:17.490743 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:40:17.490710 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:40:17.556199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:17.556167 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:17.672960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:17.672928 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:18.495329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:18.495292 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:23.105032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:23.105002 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:23.247218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:23.247187 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:24.020688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:24.020654 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:40:27.819493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:27.819437 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:28.194443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:28.194355 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:28.978000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:28.977966 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:29.580657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:29.580620 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:29.830637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:29.830593 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:32.985168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:32.985136 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:33.139802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:33.139758 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:33.266862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:33.266778 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:33.705895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:33.705813 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:33.872533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:33.872495 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:34.005943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:34.005910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:35.096849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:35.096816 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:36.193076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.193045 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66"] Oct 31 15:40:36.305209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.305181 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c"] Oct 31 15:40:36.537137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.537080 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:36.661568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.661509 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:36.825076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.825001 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl"] Oct 31 15:40:36.825711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.825695 4865 cpu_manager.go:414] "containerMap: removing container" podUID="979de514-5322-4324-9a94-46f79c2e6c65" containerName="main" Oct 31 15:40:36.825770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.825713 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979de514-5322-4324-9a94-46f79c2e6c65" containerName="main" Oct 31 15:40:36.825869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.825860 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="979de514-5322-4324-9a94-46f79c2e6c65" containerName="main" Oct 31 15:40:36.829551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.829535 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877270 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-data\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877329 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05802775-dff2-4639-b895-2cbae3b5d52e-istiod-ca-cert\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877537 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmm4\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-kube-api-access-lnmm4\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877578 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-certs\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877657 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-envoy\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877715 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-credential-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877759 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05802775-dff2-4639-b895-2cbae3b5d52e-istio-podinfo\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877832 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.877977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.877898 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-istio-token\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.883959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.883936 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-24ca6e-openshift-default-dockercfg-848vm\"" Oct 31 15:40:36.887259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.887234 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl"] Oct 31 15:40:36.979066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979038 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-data\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979104 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05802775-dff2-4639-b895-2cbae3b5d52e-istiod-ca-cert\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979179 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmm4\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-kube-api-access-lnmm4\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979203 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-certs\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979236 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-envoy\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979266 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-credential-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979339 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05802775-dff2-4639-b895-2cbae3b5d52e-istio-podinfo\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979417 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979505 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-data\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979507 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-istio-token\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979689 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-certs\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979774 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979792 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-credential-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.979955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.979928 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05802775-dff2-4639-b895-2cbae3b5d52e-istiod-ca-cert\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.981901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.981877 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-envoy\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:36.982143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:36.982123 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05802775-dff2-4639-b895-2cbae3b5d52e-istio-podinfo\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:37.037184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:37.037160 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-istio-token\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:37.037320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:37.037231 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmm4\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-kube-api-access-lnmm4\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:37.143774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:37.143691 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:37.597766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:37.574151 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl"] Oct 31 15:40:37.638495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:37.631006 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:40:37.638495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:37.631105 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:40:37.638495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:37.631151 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:40:38.287794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:38.287742 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:38.370284 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:38.370251 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" event={"ID":"05802775-dff2-4639-b895-2cbae3b5d52e","Type":"ContainerStarted","Data":"3b7f8582faa8a4a37a62134cd55b8af285709cae999f5f753d6c8514b63a4d5a"} Oct 31 15:40:38.370522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:38.370311 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" event={"ID":"05802775-dff2-4639-b895-2cbae3b5d52e","Type":"ContainerStarted","Data":"52b40a9f31356e82bed13903c2208b1f4c015b7c2d92d86b710bcf936ef3d92b"} Oct 31 15:40:38.491548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:38.491481 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podStartSLOduration=2.491451282 podStartE2EDuration="2.491451282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:40:38.488965767 +0000 UTC m=+3284.108517188" watchObservedRunningTime="2026-10-31 15:40:38.491451282 +0000 UTC m=+3284.111002692" Oct 31 15:40:39.144959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:39.144917 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:39.146317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:39.146293 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 31 15:40:39.146390 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:39.146342 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 31 15:40:39.911033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:39.910998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:40.144962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.144923 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 31 15:40:40.145365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.144985 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 31 15:40:40.155747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.155699 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:40.323588 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.323556 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d"] Oct 31 15:40:40.329094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.329070 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.346102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.346080 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da020dc-38f5-4282-8db7-3d88e9bb1f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.346222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.346200 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshwk\" (UniqueName: \"kubernetes.io/projected/1da020dc-38f5-4282-8db7-3d88e9bb1f85-kube-api-access-bshwk\") pod \"payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.382713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.382686 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-06ec34-dockercfg-hqqvz\"" Oct 31 15:40:40.384020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.383997 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d"] Oct 31 15:40:40.430815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.430780 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-06ec34\"" Oct 31 15:40:40.478495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.456392 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da020dc-38f5-4282-8db7-3d88e9bb1f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.510871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.510815 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshwk\" (UniqueName: \"kubernetes.io/projected/1da020dc-38f5-4282-8db7-3d88e9bb1f85-kube-api-access-bshwk\") pod \"payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.538635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.490105 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da020dc-38f5-4282-8db7-3d88e9bb1f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.544522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.487906 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2"] Oct 31 15:40:40.559195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.559159 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:40.584687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.584622 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2"] Oct 31 15:40:40.624553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.624370 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshwk\" (UniqueName: \"kubernetes.io/projected/1da020dc-38f5-4282-8db7-3d88e9bb1f85-kube-api-access-bshwk\") pod \"payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.644652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.644623 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:40.646131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.646106 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhvn\" (UniqueName: \"kubernetes.io/projected/6a0480b5-63c2-4aca-9a4c-76741200a6a7-kube-api-access-gqhvn\") pod \"payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:40.646205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.646187 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0480b5-63c2-4aca-9a4c-76741200a6a7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:40.750495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.749396 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhvn\" (UniqueName: \"kubernetes.io/projected/6a0480b5-63c2-4aca-9a4c-76741200a6a7-kube-api-access-gqhvn\") pod \"payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:40.750495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.749510 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0480b5-63c2-4aca-9a4c-76741200a6a7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:40.750495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.750350 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0480b5-63c2-4aca-9a4c-76741200a6a7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:40.821061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.821021 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhvn\" (UniqueName: \"kubernetes.io/projected/6a0480b5-63c2-4aca-9a4c-76741200a6a7-kube-api-access-gqhvn\") pod \"payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:40.883987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:40.883911 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:41.022172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.022083 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:41.055304 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:40:41.055263 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da020dc_38f5_4282_8db7_3d88e9bb1f85.slice/crio-c317fc479951bd50185050a14880b438d1d62cce8b2cdde03f72f097407d56d0 WatchSource:0}: Error finding container c317fc479951bd50185050a14880b438d1d62cce8b2cdde03f72f097407d56d0: Status 404 returned error can't find the container with id c317fc479951bd50185050a14880b438d1d62cce8b2cdde03f72f097407d56d0 Oct 31 15:40:41.055946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.055923 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d"] Oct 31 15:40:41.144077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.144051 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 31 15:40:41.144237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.144108 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 31 15:40:41.218727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.218675 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" podUID="865c079d-7de8-4e62-98b6-4f06f5cef32e" containerName="payload-processing" containerID="cri-o://dd6015fd706046ca848dad8dff06175ac87eae91bce648b4d68851aaddc33b18" gracePeriod=25 Oct 31 15:40:41.278858 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:40:41.278825 4865 logging.go:55] [core] [Channel #9979 SubChannel #9981]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.36:50052", ServerName: "10.132.1.36:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.36:50052: connect: connection refused" Oct 31 15:40:41.286638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.286613 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2"] Oct 31 15:40:41.290625 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:40:41.290602 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0480b5_63c2_4aca_9a4c_76741200a6a7.slice/crio-d12b7011c5b01e43c82f7507508c587d371421bcfb68d843f7c5d6e7735a4bb5 WatchSource:0}: Error finding container d12b7011c5b01e43c82f7507508c587d371421bcfb68d843f7c5d6e7735a4bb5: Status 404 returned error can't find the container with id d12b7011c5b01e43c82f7507508c587d371421bcfb68d843f7c5d6e7735a4bb5 Oct 31 15:40:41.317494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.306542 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" podUID="38ed24e4-2bc0-4510-acba-e13bcac34b49" containerName="payload-pre-processing" containerID="cri-o://28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34" gracePeriod=25 Oct 31 15:40:41.432355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.432215 4865 generic.go:356] "Generic (PLEG): container finished" podID="865c079d-7de8-4e62-98b6-4f06f5cef32e" containerID="dd6015fd706046ca848dad8dff06175ac87eae91bce648b4d68851aaddc33b18" exitCode=0 Oct 31 15:40:41.432355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.432334 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" event={"ID":"865c079d-7de8-4e62-98b6-4f06f5cef32e","Type":"ContainerDied","Data":"dd6015fd706046ca848dad8dff06175ac87eae91bce648b4d68851aaddc33b18"} Oct 31 15:40:41.437903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.436237 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" event={"ID":"1da020dc-38f5-4282-8db7-3d88e9bb1f85","Type":"ContainerStarted","Data":"35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320"} Oct 31 15:40:41.437903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.436292 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" event={"ID":"1da020dc-38f5-4282-8db7-3d88e9bb1f85","Type":"ContainerStarted","Data":"c317fc479951bd50185050a14880b438d1d62cce8b2cdde03f72f097407d56d0"} Oct 31 15:40:41.437903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.436997 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:41.439564 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.439008 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" event={"ID":"6a0480b5-63c2-4aca-9a4c-76741200a6a7","Type":"ContainerStarted","Data":"d12b7011c5b01e43c82f7507508c587d371421bcfb68d843f7c5d6e7735a4bb5"} Oct 31 15:40:41.538434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.538405 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:40:41.560928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.560879 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" podStartSLOduration=1.560863799 podStartE2EDuration="1.560863799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:40:41.555856014 +0000 UTC m=+3287.175407426" watchObservedRunningTime="2026-10-31 15:40:41.560863799 +0000 UTC m=+3287.180415211" Oct 31 15:40:41.666645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.664592 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865c079d-7de8-4e62-98b6-4f06f5cef32e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865c079d-7de8-4e62-98b6-4f06f5cef32e-plugins-config-volume\") pod \"865c079d-7de8-4e62-98b6-4f06f5cef32e\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " Oct 31 15:40:41.666645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.664829 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865c079d-7de8-4e62-98b6-4f06f5cef32e-kube-api-access-nwm2z\" (UniqueName: \"kubernetes.io/projected/865c079d-7de8-4e62-98b6-4f06f5cef32e-kube-api-access-nwm2z\") pod \"865c079d-7de8-4e62-98b6-4f06f5cef32e\" (UID: \"865c079d-7de8-4e62-98b6-4f06f5cef32e\") " Oct 31 15:40:41.666645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.665824 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865c079d-7de8-4e62-98b6-4f06f5cef32e-plugins-config-volume" pod "865c079d-7de8-4e62-98b6-4f06f5cef32e" (UID: "865c079d-7de8-4e62-98b6-4f06f5cef32e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:40:41.666645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.666599 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865c079d-7de8-4e62-98b6-4f06f5cef32e-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:40:41.670419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.670360 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865c079d-7de8-4e62-98b6-4f06f5cef32e-kube-api-access-nwm2z" pod "865c079d-7de8-4e62-98b6-4f06f5cef32e" (UID: "865c079d-7de8-4e62-98b6-4f06f5cef32e"). InnerVolumeSpecName "kube-api-access-nwm2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:40:41.730457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.726909 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:40:41.767247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.767216 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ed24e4-2bc0-4510-acba-e13bcac34b49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed24e4-2bc0-4510-acba-e13bcac34b49-plugins-config-volume\") pod \"38ed24e4-2bc0-4510-acba-e13bcac34b49\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " Oct 31 15:40:41.767400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.767334 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ed24e4-2bc0-4510-acba-e13bcac34b49-kube-api-access-xt2j5\" (UniqueName: \"kubernetes.io/projected/38ed24e4-2bc0-4510-acba-e13bcac34b49-kube-api-access-xt2j5\") pod \"38ed24e4-2bc0-4510-acba-e13bcac34b49\" (UID: \"38ed24e4-2bc0-4510-acba-e13bcac34b49\") " Oct 31 15:40:41.767675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.767649 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ed24e4-2bc0-4510-acba-e13bcac34b49-plugins-config-volume" pod "38ed24e4-2bc0-4510-acba-e13bcac34b49" (UID: "38ed24e4-2bc0-4510-acba-e13bcac34b49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:40:41.767834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.767819 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ed24e4-2bc0-4510-acba-e13bcac34b49-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:40:41.767877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.767838 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwm2z\" (UniqueName: \"kubernetes.io/projected/865c079d-7de8-4e62-98b6-4f06f5cef32e-kube-api-access-nwm2z\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:40:41.769687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.769664 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ed24e4-2bc0-4510-acba-e13bcac34b49-kube-api-access-xt2j5" pod "38ed24e4-2bc0-4510-acba-e13bcac34b49" (UID: "38ed24e4-2bc0-4510-acba-e13bcac34b49"). InnerVolumeSpecName "kube-api-access-xt2j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:40:41.869832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:41.869794 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2j5\" (UniqueName: \"kubernetes.io/projected/38ed24e4-2bc0-4510-acba-e13bcac34b49-kube-api-access-xt2j5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:40:42.145212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.145178 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 31 15:40:42.145401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.145231 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 31 15:40:42.278984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.278946 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" podUID="865c079d-7de8-4e62-98b6-4f06f5cef32e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.36:50052\" within 1s: context deadline exceeded" Oct 31 15:40:42.447515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.447411 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" Oct 31 15:40:42.447515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.447442 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66" event={"ID":"865c079d-7de8-4e62-98b6-4f06f5cef32e","Type":"ContainerDied","Data":"1e4912e84f35377f33f2d8aeed1518b05924ad1fd3a17a366d75123ec4e01fe7"} Oct 31 15:40:42.447757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.447546 4865 scope.go:122] "RemoveContainer" containerID="dd6015fd706046ca848dad8dff06175ac87eae91bce648b4d68851aaddc33b18" Oct 31 15:40:42.449030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.449009 4865 generic.go:356] "Generic (PLEG): container finished" podID="38ed24e4-2bc0-4510-acba-e13bcac34b49" containerID="28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34" exitCode=0 Oct 31 15:40:42.449142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.449059 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" Oct 31 15:40:42.449142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.449094 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" event={"ID":"38ed24e4-2bc0-4510-acba-e13bcac34b49","Type":"ContainerDied","Data":"28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34"} Oct 31 15:40:42.449142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.449128 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c" event={"ID":"38ed24e4-2bc0-4510-acba-e13bcac34b49","Type":"ContainerDied","Data":"4ef99425ceab0eb96d510ef1e8fcf94442965d85faf0d9705189ec6f2e8da2c7"} Oct 31 15:40:42.451005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.450982 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" event={"ID":"6a0480b5-63c2-4aca-9a4c-76741200a6a7","Type":"ContainerStarted","Data":"c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a"} Oct 31 15:40:42.451127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.451051 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:42.459431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.459410 4865 scope.go:122] "RemoveContainer" containerID="28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34" Oct 31 15:40:42.468594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.468577 4865 scope.go:122] "RemoveContainer" containerID="28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34" Oct 31 15:40:42.468868 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:40:42.468843 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34\": container with ID starting with 28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34 not found: ID does not exist" containerID="28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34" Oct 31 15:40:42.468924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.468878 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34"} err="failed to get container status \"28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34\": rpc error: code = NotFound desc = could not find container \"28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34\": container with ID starting with 28c7d9dde58f6ee863bc0b3de1568af01bfd910c4df976ebd46f1fd303304d34 not found: ID does not exist" Oct 31 15:40:42.626021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.625964 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" podStartSLOduration=2.625948133 podStartE2EDuration="2.625948133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:40:42.624884569 +0000 UTC m=+3288.244435978" watchObservedRunningTime="2026-10-31 15:40:42.625948133 +0000 UTC m=+3288.245499593" Oct 31 15:40:42.750348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.750313 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c"] Oct 31 15:40:42.754751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.754724 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2cb1b7-7bfc585fc5-n6h2c"] Oct 31 15:40:42.869038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.869004 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66"] Oct 31 15:40:42.924859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:42.924823 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cb1b7-c6d5cc55f-7lh66"] Oct 31 15:40:43.144196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:43.144101 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Oct 31 15:40:43.144196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:43.144159 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Oct 31 15:40:43.234834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:43.234797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:44.390896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:44.390863 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:44.391456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:44.391425 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:44.392494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:44.392445 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:40:44.497931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:44.497904 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ed24e4-2bc0-4510-acba-e13bcac34b49" path="/var/lib/kubelet/pods/38ed24e4-2bc0-4510-acba-e13bcac34b49/volumes" Oct 31 15:40:44.498324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:44.498309 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865c079d-7de8-4e62-98b6-4f06f5cef32e" path="/var/lib/kubelet/pods/865c079d-7de8-4e62-98b6-4f06f5cef32e/volumes" Oct 31 15:40:47.491043 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:40:47.490997 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:40:47.886962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:47.886928 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:52.455012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:52.454980 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:40:52.599043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:52.598992 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:53.463729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:53.463700 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:40:56.031346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:56.031317 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:40:56.039413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:56.039390 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:40:56.968739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:56.968702 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:40:57.331053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:57.330979 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:40:58.472377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:58.472344 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:40:58.480815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:40:58.480795 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:41:01.366276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:01.366235 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:03.065456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:03.065420 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:03.762794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:03.762752 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:03.893568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:03.893517 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:05.221278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:05.221214 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:05.350046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:05.350009 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:05.778668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:05.778630 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:07.676721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:07.676633 4865 generic.go:356] "Generic (PLEG): container finished" podID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerID="781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f" exitCode=0 Oct 31 15:41:07.676721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:07.676703 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" event={"ID":"c9036291-7198-4aca-8a3d-84ef77b0705f","Type":"ContainerDied","Data":"781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f"} Oct 31 15:41:07.677340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:07.677250 4865 scope.go:122] "RemoveContainer" containerID="781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f" Oct 31 15:41:07.679600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:07.679532 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:41:07.679722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:07.679610 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:41:07.679722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:07.679644 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:41:08.029073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:08.029032 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:08.688009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:08.687973 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" event={"ID":"c9036291-7198-4aca-8a3d-84ef77b0705f","Type":"ContainerStarted","Data":"a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9"} Oct 31 15:41:09.514495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:09.505842 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:41:09.519367 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:09.519338 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-c59395-openshift-default-547997b685-sd645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 31 15:41:09.519579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:09.519397 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 31 15:41:09.988369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:09.988329 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:10.510494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:10.493539 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-c59395-openshift-default-547997b685-sd645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 31 15:41:10.510494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:10.493644 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 31 15:41:10.684499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:10.680587 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:41:10.924129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:10.924033 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:11.232911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:11.232877 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:11.506555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:11.497871 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-c59395-openshift-default-547997b685-sd645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 31 15:41:11.506555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:11.497935 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 31 15:41:12.493567 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:12.493522 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-c59395-openshift-default-547997b685-sd645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 31 15:41:12.494089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:12.493595 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 31 15:41:13.213496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:13.211063 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:13.516551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:13.498820 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-c59395-openshift-default-547997b685-sd645 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Oct 31 15:41:13.516551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:13.498885 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Oct 31 15:41:14.499546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:14.499517 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:41:14.500583 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:14.500559 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:41:14.955303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:14.955224 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:15.143809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:15.143778 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:15.278667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:15.278632 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:17.491111 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:41:17.491063 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:41:18.330077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:18.330044 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:18.777153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:18.777112 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:18.945199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:18.945159 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:19.192562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.192479 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h"] Oct 31 15:41:19.193204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.193191 4865 cpu_manager.go:414] "containerMap: removing container" podUID="865c079d-7de8-4e62-98b6-4f06f5cef32e" containerName="payload-processing" Oct 31 15:41:19.193250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.193206 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865c079d-7de8-4e62-98b6-4f06f5cef32e" containerName="payload-processing" Oct 31 15:41:19.193250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.193224 4865 cpu_manager.go:414] "containerMap: removing container" podUID="38ed24e4-2bc0-4510-acba-e13bcac34b49" containerName="payload-pre-processing" Oct 31 15:41:19.193250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.193230 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ed24e4-2bc0-4510-acba-e13bcac34b49" containerName="payload-pre-processing" Oct 31 15:41:19.193360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.193331 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="865c079d-7de8-4e62-98b6-4f06f5cef32e" containerName="payload-processing" Oct 31 15:41:19.193360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.193341 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ed24e4-2bc0-4510-acba-e13bcac34b49" containerName="payload-pre-processing" Oct 31 15:41:19.198252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.198222 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.252099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.252069 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h"] Oct 31 15:41:19.306493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.305772 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5a0cef-dockercfg-vfqms\"" Oct 31 15:41:19.306493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.306421 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5a0cef\"" Oct 31 15:41:19.330493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.314386 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfrp\" (UniqueName: \"kubernetes.io/projected/3068771f-a256-41f8-b2f2-84c9738c8408-kube-api-access-4pfrp\") pod \"payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.330904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.330875 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3068771f-a256-41f8-b2f2-84c9738c8408-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.364794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.364758 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh"] Oct 31 15:41:19.371250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.371221 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.410005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.409977 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh"] Oct 31 15:41:19.432436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.432404 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkc65\" (UniqueName: \"kubernetes.io/projected/b7498c45-6d3a-44e0-a70e-92b62b937315-kube-api-access-gkc65\") pod \"payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.432629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.432611 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfrp\" (UniqueName: \"kubernetes.io/projected/3068771f-a256-41f8-b2f2-84c9738c8408-kube-api-access-4pfrp\") pod \"payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.432722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.432709 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7498c45-6d3a-44e0-a70e-92b62b937315-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.432766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.432747 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3068771f-a256-41f8-b2f2-84c9738c8408-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.434128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.434103 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3068771f-a256-41f8-b2f2-84c9738c8408-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.490270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.490228 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfrp\" (UniqueName: \"kubernetes.io/projected/3068771f-a256-41f8-b2f2-84c9738c8408-kube-api-access-4pfrp\") pod \"payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.513246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.513221 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:19.534463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.534431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7498c45-6d3a-44e0-a70e-92b62b937315-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.534678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.534658 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkc65\" (UniqueName: \"kubernetes.io/projected/b7498c45-6d3a-44e0-a70e-92b62b937315-kube-api-access-gkc65\") pod \"payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.535223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.535182 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7498c45-6d3a-44e0-a70e-92b62b937315-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.594275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.594238 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkc65\" (UniqueName: \"kubernetes.io/projected/b7498c45-6d3a-44e0-a70e-92b62b937315-kube-api-access-gkc65\") pod \"payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.686906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.686882 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:19.890115 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:41:19.890075 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3068771f_a256_41f8_b2f2_84c9738c8408.slice/crio-9802bd8877770eb03bc13b64bd6e38363415cf17b3500a62ff4619e2511d61a2 WatchSource:0}: Error finding container 9802bd8877770eb03bc13b64bd6e38363415cf17b3500a62ff4619e2511d61a2: Status 404 returned error can't find the container with id 9802bd8877770eb03bc13b64bd6e38363415cf17b3500a62ff4619e2511d61a2 Oct 31 15:41:19.890695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:19.890669 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h"] Oct 31 15:41:20.061447 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:41:20.061414 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7498c45_6d3a_44e0_a70e_92b62b937315.slice/crio-637141c71491d8b9473eb8403bc3071f677327bb15f4f2923a897854dfd54de7 WatchSource:0}: Error finding container 637141c71491d8b9473eb8403bc3071f677327bb15f4f2923a897854dfd54de7: Status 404 returned error can't find the container with id 637141c71491d8b9473eb8403bc3071f677327bb15f4f2923a897854dfd54de7 Oct 31 15:41:20.061998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.061969 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh"] Oct 31 15:41:20.822916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.822876 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" event={"ID":"b7498c45-6d3a-44e0-a70e-92b62b937315","Type":"ContainerStarted","Data":"7fbc74a4a2f1fc99965e9fc653e2cfeec3e69a2d8b929bb365edffb57820baf2"} Oct 31 15:41:20.823096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.822981 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" event={"ID":"b7498c45-6d3a-44e0-a70e-92b62b937315","Type":"ContainerStarted","Data":"637141c71491d8b9473eb8403bc3071f677327bb15f4f2923a897854dfd54de7"} Oct 31 15:41:20.823175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.823148 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:20.824631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.824608 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" event={"ID":"3068771f-a256-41f8-b2f2-84c9738c8408","Type":"ContainerStarted","Data":"584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6"} Oct 31 15:41:20.824760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.824643 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" event={"ID":"3068771f-a256-41f8-b2f2-84c9738c8408","Type":"ContainerStarted","Data":"9802bd8877770eb03bc13b64bd6e38363415cf17b3500a62ff4619e2511d61a2"} Oct 31 15:41:20.824760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.824682 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:20.940255 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:20.940211 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" podStartSLOduration=1.940199547 podStartE2EDuration="1.940199547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:20.937172505 +0000 UTC m=+3326.556723917" watchObservedRunningTime="2026-10-31 15:41:20.940199547 +0000 UTC m=+3326.559750955" Oct 31 15:41:21.054694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:21.054628 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" podStartSLOduration=2.054609 podStartE2EDuration="2.054609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:21.050830144 +0000 UTC m=+3326.670381558" watchObservedRunningTime="2026-10-31 15:41:21.054609 +0000 UTC m=+3326.674160413" Oct 31 15:41:23.316484 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:23.316435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:28.208489 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:28.208439 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:30.047194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:30.047155 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q"] Oct 31 15:41:30.097779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:30.097744 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ee73bf-2n47q"] Oct 31 15:41:30.497780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:30.497750 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b59b90d-672d-4acc-93f3-2d238f9e4829" path="/var/lib/kubelet/pods/0b59b90d-672d-4acc-93f3-2d238f9e4829/volumes" Oct 31 15:41:31.835433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:31.835404 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:41:31.835926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:31.835893 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:41:32.922857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:32.922819 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:37.162039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:37.162005 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:37.847960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:37.847926 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:38.466125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:38.466090 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:39.061968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:39.061874 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:39.327917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:39.327836 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:41.047832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.047790 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6"] Oct 31 15:41:41.101267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.101228 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683e84c-dzsv6"] Oct 31 15:41:41.995730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.995698 4865 generic.go:356] "Generic (PLEG): container finished" podID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerID="9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843" exitCode=0 Oct 31 15:41:41.995948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.995772 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" event={"ID":"c9e3f80f-a50e-4b21-87da-4e069cf41144","Type":"ContainerDied","Data":"9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843"} Oct 31 15:41:41.996183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.996163 4865 scope.go:122] "RemoveContainer" containerID="9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843" Oct 31 15:41:41.997815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.997786 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:41:41.997882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.997863 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:41:41.997923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:41.997897 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:41:42.498407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:42.498378 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6422ef93-38ad-49f9-8609-e6b0e88b9a4e" path="/var/lib/kubelet/pods/6422ef93-38ad-49f9-8609-e6b0e88b9a4e/volumes" Oct 31 15:41:42.521495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:42.519056 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:43.025315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:43.025279 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" event={"ID":"c9e3f80f-a50e-4b21-87da-4e069cf41144","Type":"ContainerStarted","Data":"845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf"} Oct 31 15:41:43.270063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:43.270026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:43.439575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:43.439496 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:43.582021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:43.581983 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:43.888103 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:43.888068 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:41:43.889602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:43.889572 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 15:41:43.889744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:43.889628 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 15:41:44.887884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:44.887850 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 15:41:44.888295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:44.887914 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 15:41:45.888149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:45.888109 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 15:41:45.888662 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:45.888182 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 15:41:46.478903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.478871 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:46.615747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.615712 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:46.862801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.862717 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v"] Oct 31 15:41:46.866078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.866060 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:46.888221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.888194 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 15:41:46.888555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.888251 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 15:41:46.923417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.923383 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v"] Oct 31 15:41:46.974654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.974622 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-69517a\"" Oct 31 15:41:46.975044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:46.975025 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-69517a-dockercfg-g7cll\"" Oct 31 15:41:47.049938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.049901 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b67d0c-a8c4-42bd-9b15-757809b4447b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:47.050379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.050353 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lfq\" (UniqueName: \"kubernetes.io/projected/d7b67d0c-a8c4-42bd-9b15-757809b4447b-kube-api-access-c5lfq\") pod \"payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:47.151769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.151684 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b67d0c-a8c4-42bd-9b15-757809b4447b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:47.151954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.151843 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lfq\" (UniqueName: \"kubernetes.io/projected/d7b67d0c-a8c4-42bd-9b15-757809b4447b-kube-api-access-c5lfq\") pod \"payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:47.152530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.152508 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b67d0c-a8c4-42bd-9b15-757809b4447b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:47.220986 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.220952 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lfq\" (UniqueName: \"kubernetes.io/projected/d7b67d0c-a8c4-42bd-9b15-757809b4447b-kube-api-access-c5lfq\") pod \"payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:47.479410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.479382 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:47.492208 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:41:47.492173 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:41:47.520523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.508560 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:41:47.755772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.755693 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:47.885235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.885208 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v"] Oct 31 15:41:47.885515 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:41:47.885492 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b67d0c_a8c4_42bd_9b15_757809b4447b.slice/crio-03770401a3b8e9d05f44659e69f4cb37876cbec3d5dcfc4b813c704d88905b24 WatchSource:0}: Error finding container 03770401a3b8e9d05f44659e69f4cb37876cbec3d5dcfc4b813c704d88905b24: Status 404 returned error can't find the container with id 03770401a3b8e9d05f44659e69f4cb37876cbec3d5dcfc4b813c704d88905b24 Oct 31 15:41:47.887780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.887756 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Oct 31 15:41:47.887903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:47.887797 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Oct 31 15:41:48.069281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:48.069180 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" event={"ID":"d7b67d0c-a8c4-42bd-9b15-757809b4447b","Type":"ContainerStarted","Data":"97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d"} Oct 31 15:41:48.069281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:48.069242 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" event={"ID":"d7b67d0c-a8c4-42bd-9b15-757809b4447b","Type":"ContainerStarted","Data":"03770401a3b8e9d05f44659e69f4cb37876cbec3d5dcfc4b813c704d88905b24"} Oct 31 15:41:48.069281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:48.069263 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:41:48.185982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:48.185915 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" podStartSLOduration=2.185893726 podStartE2EDuration="2.185893726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:48.183020655 +0000 UTC m=+3353.802572077" watchObservedRunningTime="2026-10-31 15:41:48.185893726 +0000 UTC m=+3353.805445138" Oct 31 15:41:48.893027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:48.892998 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:41:48.894234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:48.894211 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:41:52.724370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:52.724341 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:52.992549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:52.992450 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:41:57.557230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:57.557198 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:57.712105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:57.712070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:57.832995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:57.832914 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:41:59.080725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:41:59.080701 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:42:02.392796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:02.392762 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:04.143751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:04.143714 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:05.563804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:05.563714 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:07.089705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:07.089672 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:07.697735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:07.697702 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:11.550127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:11.550085 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:12.139080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:12.139018 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:16.651043 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:16.651009 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc"] Oct 31 15:42:16.760186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:16.760153 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf"] Oct 31 15:42:17.034427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:17.034387 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:17.161749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:17.161721 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:17.336745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:17.336655 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:17.492210 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:42:17.492168 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:42:17.531442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:17.531409 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:21.651760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:21.651724 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" podUID="67f3fbff-e845-47e8-a338-eb89c17e9ba5" containerName="payload-processing" containerID="cri-o://3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031" gracePeriod=25 Oct 31 15:42:21.760477 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:21.760410 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" podUID="12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" containerName="payload-pre-processing" containerID="cri-o://719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2" gracePeriod=25 Oct 31 15:42:21.931732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:21.931702 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:42:22.012014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.011974 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f3fbff-e845-47e8-a338-eb89c17e9ba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f3fbff-e845-47e8-a338-eb89c17e9ba5-plugins-config-volume\") pod \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " Oct 31 15:42:22.012205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.012098 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f3fbff-e845-47e8-a338-eb89c17e9ba5-kube-api-access-b2bhs\" (UniqueName: \"kubernetes.io/projected/67f3fbff-e845-47e8-a338-eb89c17e9ba5-kube-api-access-b2bhs\") pod \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\" (UID: \"67f3fbff-e845-47e8-a338-eb89c17e9ba5\") " Oct 31 15:42:22.012368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.012339 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f3fbff-e845-47e8-a338-eb89c17e9ba5-plugins-config-volume" pod "67f3fbff-e845-47e8-a338-eb89c17e9ba5" (UID: "67f3fbff-e845-47e8-a338-eb89c17e9ba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:42:22.012595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.012579 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f3fbff-e845-47e8-a338-eb89c17e9ba5-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:42:22.014391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.014366 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f3fbff-e845-47e8-a338-eb89c17e9ba5-kube-api-access-b2bhs" pod "67f3fbff-e845-47e8-a338-eb89c17e9ba5" (UID: "67f3fbff-e845-47e8-a338-eb89c17e9ba5"). InnerVolumeSpecName "kube-api-access-b2bhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:42:22.047443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.047422 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:42:22.113681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.113653 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-plugins-config-volume\") pod \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " Oct 31 15:42:22.113842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.113772 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-kube-api-access-lvgq2\" (UniqueName: \"kubernetes.io/projected/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-kube-api-access-lvgq2\") pod \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\" (UID: \"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e\") " Oct 31 15:42:22.114105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.114076 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-plugins-config-volume" pod "12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" (UID: "12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:42:22.114377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.114359 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:42:22.114430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.114381 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2bhs\" (UniqueName: \"kubernetes.io/projected/67f3fbff-e845-47e8-a338-eb89c17e9ba5-kube-api-access-b2bhs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:42:22.116006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.115984 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-kube-api-access-lvgq2" pod "12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" (UID: "12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e"). InnerVolumeSpecName "kube-api-access-lvgq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:42:22.215321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.215294 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvgq2\" (UniqueName: \"kubernetes.io/projected/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e-kube-api-access-lvgq2\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:42:22.318884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.318849 4865 generic.go:356] "Generic (PLEG): container finished" podID="12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" containerID="719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2" exitCode=0 Oct 31 15:42:22.319061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.318891 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" Oct 31 15:42:22.319061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.318933 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" event={"ID":"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e","Type":"ContainerDied","Data":"719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2"} Oct 31 15:42:22.319061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.318989 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf" event={"ID":"12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e","Type":"ContainerDied","Data":"6233380670f226339520a3b274488527cc63457eba9893ee83c9a0093c2bc3ad"} Oct 31 15:42:22.319061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.319021 4865 scope.go:122] "RemoveContainer" containerID="719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2" Oct 31 15:42:22.320878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.320855 4865 generic.go:356] "Generic (PLEG): container finished" podID="67f3fbff-e845-47e8-a338-eb89c17e9ba5" containerID="3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031" exitCode=0 Oct 31 15:42:22.321098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.320904 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" event={"ID":"67f3fbff-e845-47e8-a338-eb89c17e9ba5","Type":"ContainerDied","Data":"3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031"} Oct 31 15:42:22.321191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.321136 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" event={"ID":"67f3fbff-e845-47e8-a338-eb89c17e9ba5","Type":"ContainerDied","Data":"f2a2275a73990411eb150d2d0ea05e9a4936a2d95304cc8f2cff0d3bc688c08a"} Oct 31 15:42:22.321191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.320916 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc" Oct 31 15:42:22.330924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.330903 4865 scope.go:122] "RemoveContainer" containerID="719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2" Oct 31 15:42:22.331348 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:42:22.331324 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2\": container with ID starting with 719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2 not found: ID does not exist" containerID="719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2" Oct 31 15:42:22.331440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.331354 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2"} err="failed to get container status \"719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2\": rpc error: code = NotFound desc = could not find container \"719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2\": container with ID starting with 719ba836b5d06b27292941dbe77d2c94281514397cc3beb000c6013593da16d2 not found: ID does not exist" Oct 31 15:42:22.331440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.331372 4865 scope.go:122] "RemoveContainer" containerID="3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031" Oct 31 15:42:22.342029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.342008 4865 scope.go:122] "RemoveContainer" containerID="3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031" Oct 31 15:42:22.342316 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:42:22.342276 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031\": container with ID starting with 3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031 not found: ID does not exist" containerID="3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031" Oct 31 15:42:22.342392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.342328 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031"} err="failed to get container status \"3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031\": rpc error: code = NotFound desc = could not find container \"3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031\": container with ID starting with 3b976f5eadbc37c513fe321ee1a605c1b7049212a99f28d6db8727b1f252a031 not found: ID does not exist" Oct 31 15:42:22.406671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.406646 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:22.487269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.487181 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf"] Oct 31 15:42:22.497963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.497932 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c59395-64d49d5f79-6xxrf"] Oct 31 15:42:22.600013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.599983 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc"] Oct 31 15:42:22.603906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.603877 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c59395-67f46d8575-zz8mc"] Oct 31 15:42:22.949389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:22.949312 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:24.525513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:24.525442 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" path="/var/lib/kubelet/pods/12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e/volumes" Oct 31 15:42:24.526279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:24.526041 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f3fbff-e845-47e8-a338-eb89c17e9ba5" path="/var/lib/kubelet/pods/67f3fbff-e845-47e8-a338-eb89c17e9ba5/volumes" Oct 31 15:42:27.572783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:27.572747 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:32.458097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:32.458067 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:37.332869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:37.332833 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:42.029041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:42.029010 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:42.197790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:42.197761 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:42.335279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:42.333229 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:45.045377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:45.045344 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd"] Oct 31 15:42:45.049149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:45.049119 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289b757e-cclpd"] Oct 31 15:42:45.096058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:45.096025 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:42:46.049144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:46.049116 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz"] Oct 31 15:42:46.099869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:46.099844 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4b7f1a-sdxvz"] Oct 31 15:42:46.505234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:46.505197 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2241d3-0dcf-4ae2-924b-f654554ea568" path="/var/lib/kubelet/pods/4a2241d3-0dcf-4ae2-924b-f654554ea568/volumes" Oct 31 15:42:46.505678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:46.505664 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce51ff42-0493-447d-9401-89804bbe8115" path="/var/lib/kubelet/pods/ce51ff42-0493-447d-9401-89804bbe8115/volumes" Oct 31 15:42:46.825016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:46.824940 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:47.491194 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:42:47.491160 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:42:51.518173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:51.518126 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:55.047314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:55.047277 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl"] Oct 31 15:42:55.097998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:55.097963 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed06859d-cbtjl"] Oct 31 15:42:56.203819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:56.203786 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:42:56.498258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:56.498222 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2d170c-c6f9-48d5-a496-8f2c9b6f0504" path="/var/lib/kubelet/pods/1d2d170c-c6f9-48d5-a496-8f2c9b6f0504/volumes" Oct 31 15:42:57.045445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:57.045417 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt"] Oct 31 15:42:57.096281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:57.096248 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb478c9f-hzjjt"] Oct 31 15:42:58.497680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:58.497651 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762367c9-5274-40e0-ac0d-32a7da37af9d" path="/var/lib/kubelet/pods/762367c9-5274-40e0-ac0d-32a7da37af9d/volumes" Oct 31 15:42:59.081881 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:59.081849 4865 scope.go:122] "RemoveContainer" containerID="b67dba7f696c88e470559c8f960f72e796c7520c9f1f1c3096300d052076b89d" Oct 31 15:42:59.092671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:42:59.092651 4865 scope.go:122] "RemoveContainer" containerID="160d5ec1fc629801aff972b17faf8bd9c0cb414a9991bfa7f6d429cf14f8369c" Oct 31 15:43:00.895044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:00.895011 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:04.486083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:04.486047 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:05.591191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:05.591148 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:05.956147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:05.956065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:10.326893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:10.326857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:14.895952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:14.895911 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:15.076882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:15.076852 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:15.206205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:15.206117 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:17.490801 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:43:17.490767 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:43:19.802901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:19.802864 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:24.466205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:24.466168 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:25.973937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:25.973902 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:43:29.122806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:29.122775 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:33.811732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:33.811697 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:37.668795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:37.668702 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:37.904800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:37.904765 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:39.750687 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:39.750650 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:41.156560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:41.156524 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:41.281674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:41.281633 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:43.046576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:43.046537 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:43.341813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:43.341737 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:43.683371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:43.683276 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:43.693709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:43.693680 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645"] Oct 31 15:43:43.693904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:43.693883 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" containerID="cri-o://a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9" gracePeriod=30 Oct 31 15:43:44.502889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:44.502852 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-c59395-openshift-default-547997b685-sd645 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:43:44.503307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:44.502904 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:43:44.729298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:44.729263 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj"] Oct 31 15:43:44.837165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:44.837087 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc"] Oct 31 15:43:44.838212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:44.838184 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:45.074154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:45.074120 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:45.204672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:45.204597 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:45.779675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:45.779637 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:45.903971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:45.903934 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:47.218677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:47.218641 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:47.342688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:47.342653 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:47.492127 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:43:47.492034 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:43:48.970114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:48.970089 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:43:48.984037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:48.984004 4865 generic.go:356] "Generic (PLEG): container finished" podID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerID="a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9" exitCode=0 Oct 31 15:43:48.984196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:48.984069 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" Oct 31 15:43:48.984196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:48.984088 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" event={"ID":"c9036291-7198-4aca-8a3d-84ef77b0705f","Type":"ContainerDied","Data":"a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9"} Oct 31 15:43:48.984196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:48.984148 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645" event={"ID":"c9036291-7198-4aca-8a3d-84ef77b0705f","Type":"ContainerDied","Data":"72e81e7ae02dc637e67a04606c8a9a8a2e324effeb25eac1459c52bbe2408466"} Oct 31 15:43:48.984196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:48.984179 4865 scope.go:122] "RemoveContainer" containerID="a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9" Oct 31 15:43:48.994858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:48.994834 4865 scope.go:122] "RemoveContainer" containerID="781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f" Oct 31 15:43:49.005213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.005191 4865 scope.go:122] "RemoveContainer" containerID="a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9" Oct 31 15:43:49.005634 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:43:49.005615 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9\": container with ID starting with a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9 not found: ID does not exist" containerID="a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9" Oct 31 15:43:49.005689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.005644 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9"} err="failed to get container status \"a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9\": rpc error: code = NotFound desc = could not find container \"a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9\": container with ID starting with a63f05181a7eff63554632bbb1550265aae84d6a04188dcc7c5879cd2bdeeff9 not found: ID does not exist" Oct 31 15:43:49.005689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.005664 4865 scope.go:122] "RemoveContainer" containerID="781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f" Oct 31 15:43:49.005918 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:43:49.005897 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f\": container with ID starting with 781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f not found: ID does not exist" containerID="781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f" Oct 31 15:43:49.005976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.005924 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f"} err="failed to get container status \"781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f\": rpc error: code = NotFound desc = could not find container \"781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f\": container with ID starting with 781dd37be5f62269d37a091d87e287f185551f933b6a7d1c62b6362facc53c8f not found: ID does not exist" Oct 31 15:43:49.007637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.007610 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:49.076601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.076563 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-podinfo\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.076812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.076629 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-token\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-token\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.076812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.076660 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9036291-7198-4aca-8a3d-84ef77b0705f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9036291-7198-4aca-8a3d-84ef77b0705f-istiod-ca-cert\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.076812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.076784 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-data\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.077006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.076860 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-credential-socket\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.077006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.076924 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-certs\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.077006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.076963 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-envoy\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.077006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077000 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-kube-api-access-q9vk5\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-kube-api-access-q9vk5\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.077297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077082 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-socket\") pod \"c9036291-7198-4aca-8a3d-84ef77b0705f\" (UID: \"c9036291-7198-4aca-8a3d-84ef77b0705f\") " Oct 31 15:43:49.077297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077086 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-data" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:43:49.077297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077152 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9036291-7198-4aca-8a3d-84ef77b0705f-istiod-ca-cert" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:43:49.077297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077185 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-certs" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:43:49.077297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077222 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-credential-socket" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:43:49.077632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077418 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-socket" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:43:49.077922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077878 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.077922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077908 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.078123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077926 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.078123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077942 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.078123 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.077956 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9036291-7198-4aca-8a3d-84ef77b0705f-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.079306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.079279 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-podinfo" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:43:49.079642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.079618 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-kube-api-access-q9vk5" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "kube-api-access-q9vk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:43:49.079717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.079642 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-envoy" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:43:49.079767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.079717 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-token" pod "c9036291-7198-4aca-8a3d-84ef77b0705f" (UID: "c9036291-7198-4aca-8a3d-84ef77b0705f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:43:49.179182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.179133 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.179182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.179178 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.179433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.179193 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9036291-7198-4aca-8a3d-84ef77b0705f-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.179433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.179207 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vk5\" (UniqueName: \"kubernetes.io/projected/c9036291-7198-4aca-8a3d-84ef77b0705f-kube-api-access-q9vk5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:49.411000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.410954 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645"] Oct 31 15:43:49.414323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.414279 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c59395-openshift-default-547997b685-sd645"] Oct 31 15:43:49.730558 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.730497 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" podUID="fc9843c1-ac33-4300-a67c-368c336625d3" containerName="payload-processing" containerID="cri-o://15542dd8096f94285494ad12d5d9aa403e36af772b42d5fc81e8125c889ba34c" gracePeriod=25 Oct 31 15:43:49.838071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.838019 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" podUID="48c77f43-fb3e-4376-9454-a8cbede3f463" containerName="payload-pre-processing" containerID="cri-o://7bc1f65312a3648c99242d954a9a4474dfbb6479e05a076147b11620a3773fe2" gracePeriod=25 Oct 31 15:43:49.917053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.916827 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:49.995557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.995524 4865 generic.go:356] "Generic (PLEG): container finished" podID="48c77f43-fb3e-4376-9454-a8cbede3f463" containerID="7bc1f65312a3648c99242d954a9a4474dfbb6479e05a076147b11620a3773fe2" exitCode=0 Oct 31 15:43:49.996004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.995601 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" event={"ID":"48c77f43-fb3e-4376-9454-a8cbede3f463","Type":"ContainerDied","Data":"7bc1f65312a3648c99242d954a9a4474dfbb6479e05a076147b11620a3773fe2"} Oct 31 15:43:49.997110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.997084 4865 generic.go:356] "Generic (PLEG): container finished" podID="fc9843c1-ac33-4300-a67c-368c336625d3" containerID="15542dd8096f94285494ad12d5d9aa403e36af772b42d5fc81e8125c889ba34c" exitCode=0 Oct 31 15:43:49.997220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:49.997159 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" event={"ID":"fc9843c1-ac33-4300-a67c-368c336625d3","Type":"ContainerDied","Data":"15542dd8096f94285494ad12d5d9aa403e36af772b42d5fc81e8125c889ba34c"} Oct 31 15:43:50.028226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.028196 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:43:50.089758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.089724 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc9843c1-ac33-4300-a67c-368c336625d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9843c1-ac33-4300-a67c-368c336625d3-plugins-config-volume\") pod \"fc9843c1-ac33-4300-a67c-368c336625d3\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " Oct 31 15:43:50.090003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.089904 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9843c1-ac33-4300-a67c-368c336625d3-kube-api-access-65cxn\" (UniqueName: \"kubernetes.io/projected/fc9843c1-ac33-4300-a67c-368c336625d3-kube-api-access-65cxn\") pod \"fc9843c1-ac33-4300-a67c-368c336625d3\" (UID: \"fc9843c1-ac33-4300-a67c-368c336625d3\") " Oct 31 15:43:50.090177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.090152 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9843c1-ac33-4300-a67c-368c336625d3-plugins-config-volume" pod "fc9843c1-ac33-4300-a67c-368c336625d3" (UID: "fc9843c1-ac33-4300-a67c-368c336625d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:43:50.090524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.090505 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc9843c1-ac33-4300-a67c-368c336625d3-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:50.094129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.094096 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9843c1-ac33-4300-a67c-368c336625d3-kube-api-access-65cxn" pod "fc9843c1-ac33-4300-a67c-368c336625d3" (UID: "fc9843c1-ac33-4300-a67c-368c336625d3"). InnerVolumeSpecName "kube-api-access-65cxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:43:50.130339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.130313 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:43:50.191775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.191731 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c77f43-fb3e-4376-9454-a8cbede3f463-kube-api-access-wwctp\" (UniqueName: \"kubernetes.io/projected/48c77f43-fb3e-4376-9454-a8cbede3f463-kube-api-access-wwctp\") pod \"48c77f43-fb3e-4376-9454-a8cbede3f463\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " Oct 31 15:43:50.191994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.191874 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c77f43-fb3e-4376-9454-a8cbede3f463-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c77f43-fb3e-4376-9454-a8cbede3f463-plugins-config-volume\") pod \"48c77f43-fb3e-4376-9454-a8cbede3f463\" (UID: \"48c77f43-fb3e-4376-9454-a8cbede3f463\") " Oct 31 15:43:50.192265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.192237 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c77f43-fb3e-4376-9454-a8cbede3f463-plugins-config-volume" pod "48c77f43-fb3e-4376-9454-a8cbede3f463" (UID: "48c77f43-fb3e-4376-9454-a8cbede3f463"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:43:50.192482 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.192448 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c77f43-fb3e-4376-9454-a8cbede3f463-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:50.192560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.192502 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65cxn\" (UniqueName: \"kubernetes.io/projected/fc9843c1-ac33-4300-a67c-368c336625d3-kube-api-access-65cxn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:50.194000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.193978 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c77f43-fb3e-4376-9454-a8cbede3f463-kube-api-access-wwctp" pod "48c77f43-fb3e-4376-9454-a8cbede3f463" (UID: "48c77f43-fb3e-4376-9454-a8cbede3f463"). InnerVolumeSpecName "kube-api-access-wwctp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:43:50.293939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.293834 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwctp\" (UniqueName: \"kubernetes.io/projected/48c77f43-fb3e-4376-9454-a8cbede3f463-kube-api-access-wwctp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:50.498013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:50.497977 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" path="/var/lib/kubelet/pods/c9036291-7198-4aca-8a3d-84ef77b0705f/volumes" Oct 31 15:43:51.007411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.007319 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" event={"ID":"48c77f43-fb3e-4376-9454-a8cbede3f463","Type":"ContainerDied","Data":"6da2d421de51d1e8cf983716e6ef488f86c5270b18e77921306d6d3e6b07702c"} Oct 31 15:43:51.007897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.007327 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc" Oct 31 15:43:51.007897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.007428 4865 scope.go:122] "RemoveContainer" containerID="7bc1f65312a3648c99242d954a9a4474dfbb6479e05a076147b11620a3773fe2" Oct 31 15:43:51.009110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.009083 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" event={"ID":"fc9843c1-ac33-4300-a67c-368c336625d3","Type":"ContainerDied","Data":"aad5f72687e7de89fd0f5202846c605915fe89e2d44a8a092bfe3eeb6b9934bb"} Oct 31 15:43:51.009300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.009272 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj" Oct 31 15:43:51.018937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.018910 4865 scope.go:122] "RemoveContainer" containerID="15542dd8096f94285494ad12d5d9aa403e36af772b42d5fc81e8125c889ba34c" Oct 31 15:43:51.175828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.175784 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc"] Oct 31 15:43:51.227607 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.227577 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-982f43-7789f69959-7ksgc"] Oct 31 15:43:51.273977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.273894 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:51.290421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.290385 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj"] Oct 31 15:43:51.293482 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.293444 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-982f43-5cbfc6f975-smmdj"] Oct 31 15:43:51.797997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.797965 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f"] Oct 31 15:43:51.798686 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798669 4865 cpu_manager.go:414] "containerMap: removing container" podUID="67f3fbff-e845-47e8-a338-eb89c17e9ba5" containerName="payload-processing" Oct 31 15:43:51.798739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798687 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f3fbff-e845-47e8-a338-eb89c17e9ba5" containerName="payload-processing" Oct 31 15:43:51.798739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798703 4865 cpu_manager.go:414] "containerMap: removing container" podUID="48c77f43-fb3e-4376-9454-a8cbede3f463" containerName="payload-pre-processing" Oct 31 15:43:51.798739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798709 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c77f43-fb3e-4376-9454-a8cbede3f463" containerName="payload-pre-processing" Oct 31 15:43:51.798739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798719 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" Oct 31 15:43:51.798739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798725 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798746 4865 cpu_manager.go:414] "containerMap: removing container" podUID="12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" containerName="payload-pre-processing" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798751 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" containerName="payload-pre-processing" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798759 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fc9843c1-ac33-4300-a67c-368c336625d3" containerName="payload-processing" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798764 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc9843c1-ac33-4300-a67c-368c336625d3" containerName="payload-processing" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798882 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f3fbff-e845-47e8-a338-eb89c17e9ba5" containerName="payload-processing" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798892 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c77f43-fb3e-4376-9454-a8cbede3f463" containerName="payload-pre-processing" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798898 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" Oct 31 15:43:51.798906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798906 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a7f5a9-9c5b-499a-a6ca-ffaf0524e96e" containerName="payload-pre-processing" Oct 31 15:43:51.799162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.798913 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc9843c1-ac33-4300-a67c-368c336625d3" containerName="payload-processing" Oct 31 15:43:51.803887 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.803870 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.808886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.808857 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-routing-overlay-volume\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.808992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.808926 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vqp\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-kube-api-access-p5vqp\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.809047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.809011 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-plugins-config-volume\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.809124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.809108 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-provider-credentials\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.858689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.857131 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 15:43:51.858689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.857326 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 15:43:51.858689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.857345 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 15:43:51.858689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.858091 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f"] Oct 31 15:43:51.904712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.904683 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5a1e181487d0\"" Oct 31 15:43:51.904910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.904776 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nlvxs\"" Oct 31 15:43:51.904910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.904835 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 15:43:51.909974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.909956 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-routing-overlay-volume\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.910060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.909992 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vqp\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-kube-api-access-p5vqp\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.910060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.910021 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-plugins-config-volume\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.910060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.910057 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-provider-credentials\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.910679 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.910657 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-routing-overlay-volume\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.910818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.910799 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-plugins-config-volume\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.912775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.912753 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-provider-credentials\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:51.968252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:51.968214 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vqp\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-kube-api-access-p5vqp\") pod \"payload-processing-external-model-7c7bd584d6-v4b7f\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:52.117747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:52.117658 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:52.497855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:52.497819 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c77f43-fb3e-4376-9454-a8cbede3f463" path="/var/lib/kubelet/pods/48c77f43-fb3e-4376-9454-a8cbede3f463/volumes" Oct 31 15:43:52.498394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:52.498371 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc9843c1-ac33-4300-a67c-368c336625d3" path="/var/lib/kubelet/pods/fc9843c1-ac33-4300-a67c-368c336625d3/volumes" Oct 31 15:43:52.564836 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:43:52.564803 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ce96e1_69de_4d52_ba58_9da649ff2532.slice/crio-9eeebb06cb3d83f5cf929f47861a7745289a664499222dad7449eed04176d401 WatchSource:0}: Error finding container 9eeebb06cb3d83f5cf929f47861a7745289a664499222dad7449eed04176d401: Status 404 returned error can't find the container with id 9eeebb06cb3d83f5cf929f47861a7745289a664499222dad7449eed04176d401 Oct 31 15:43:52.564836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:52.564806 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f"] Oct 31 15:43:53.031158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:53.031117 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" event={"ID":"40ce96e1-69de-4d52-ba58-9da649ff2532","Type":"ContainerStarted","Data":"eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7"} Oct 31 15:43:53.031350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:53.031179 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" event={"ID":"40ce96e1-69de-4d52-ba58-9da649ff2532","Type":"ContainerStarted","Data":"9eeebb06cb3d83f5cf929f47861a7745289a664499222dad7449eed04176d401"} Oct 31 15:43:53.031350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:53.031217 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:43:53.148598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:53.148540 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" podStartSLOduration=2.1485216 podStartE2EDuration="2.1485216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:43:53.145807738 +0000 UTC m=+3478.765359159" watchObservedRunningTime="2026-10-31 15:43:53.1485216 +0000 UTC m=+3478.768073012" Oct 31 15:43:54.713925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:54.713875 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:43:59.135317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:59.135285 4865 scope.go:122] "RemoveContainer" containerID="19c72823044b9b6023295d8befb48a86a011765aeb78487698b203b28c3dfb6b" Oct 31 15:43:59.145774 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:59.145749 4865 scope.go:122] "RemoveContainer" containerID="e6854fbdf7bdf12d186b3c0f6cddd83468ca8207312c6704e74c0eceb049e7d8" Oct 31 15:43:59.155353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:59.155331 4865 scope.go:122] "RemoveContainer" containerID="1967680107eb16024aeaee65f2196a69273f6cb459e19ca79dd10da8e7c7bdc2" Oct 31 15:43:59.165162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:59.165141 4865 scope.go:122] "RemoveContainer" containerID="addf73dd43da7f71d1f9a7fabde3865627c19c3fb489b35367c8ca0a2352b9f1" Oct 31 15:43:59.400105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:43:59.400026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:04.040641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:04.040612 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:44:04.262931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:04.262901 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:04.832214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:04.832174 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:06.305143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:06.305112 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:08.918314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:08.918281 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:09.846535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:09.846500 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:44:13.591481 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:13.591435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:17.491675 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:17.491640 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:44:18.300485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:18.300442 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:18.618335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:18.618234 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:21.114794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.114760 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v"] Oct 31 15:44:21.115441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.115426 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" Oct 31 15:44:21.115524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.115442 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" Oct 31 15:44:21.115585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.115574 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9036291-7198-4aca-8a3d-84ef77b0705f" containerName="istio-proxy" Oct 31 15:44:21.120785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.120767 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.174137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.174106 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v"] Oct 31 15:44:21.220938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.220909 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:44:21.220938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.220910 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:44:21.221252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.221232 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:44:21.224675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.224655 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:44:21.283767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.283724 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.283933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.283853 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.283933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.283888 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsm7k\" (UniqueName: \"kubernetes.io/projected/cd01799a-fe39-4b3c-a57f-612e7c897c33-kube-api-access-wsm7k\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.385827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.385710 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.385827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.385782 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsm7k\" (UniqueName: \"kubernetes.io/projected/cd01799a-fe39-4b3c-a57f-612e7c897c33-kube-api-access-wsm7k\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.386101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.385994 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.386666 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.386644 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.388361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.388338 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.443918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.443882 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsm7k\" (UniqueName: \"kubernetes.io/projected/cd01799a-fe39-4b3c-a57f-612e7c897c33-kube-api-access-wsm7k\") pod \"maas-api-revoke-sub-bd58f104-wkt2v\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:21.734614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:21.734584 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:22.110640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:22.110611 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v"] Oct 31 15:44:22.110971 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:44:22.110949 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd01799a_fe39_4b3c_a57f_612e7c897c33.slice/crio-f4eeb5152acf5e688ff8afc840bd693ce9d4c80be6040db16f371c4d6efc8b15 WatchSource:0}: Error finding container f4eeb5152acf5e688ff8afc840bd693ce9d4c80be6040db16f371c4d6efc8b15: Status 404 returned error can't find the container with id f4eeb5152acf5e688ff8afc840bd693ce9d4c80be6040db16f371c4d6efc8b15 Oct 31 15:44:22.231183 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:22.231148 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" event={"ID":"cd01799a-fe39-4b3c-a57f-612e7c897c33","Type":"ContainerStarted","Data":"f4eeb5152acf5e688ff8afc840bd693ce9d4c80be6040db16f371c4d6efc8b15"} Oct 31 15:44:23.239077 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:23.238978 4865 generic.go:356] "Generic (PLEG): container finished" podID="cd01799a-fe39-4b3c-a57f-612e7c897c33" containerID="b20c479d329fd741b31260a2aedde2406629479cced86c9706001fbb44c15d3d" exitCode=0 Oct 31 15:44:23.239519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:23.239072 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" event={"ID":"cd01799a-fe39-4b3c-a57f-612e7c897c33","Type":"ContainerDied","Data":"b20c479d329fd741b31260a2aedde2406629479cced86c9706001fbb44c15d3d"} Oct 31 15:44:23.354904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:23.354865 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:24.399568 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.399544 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:24.530332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.530249 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-openshift-ca\") pod \"cd01799a-fe39-4b3c-a57f-612e7c897c33\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " Oct 31 15:44:24.530520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.530333 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-xks-ca\") pod \"cd01799a-fe39-4b3c-a57f-612e7c897c33\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " Oct 31 15:44:24.530520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.530382 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd01799a-fe39-4b3c-a57f-612e7c897c33-kube-api-access-wsm7k\" (UniqueName: \"kubernetes.io/projected/cd01799a-fe39-4b3c-a57f-612e7c897c33-kube-api-access-wsm7k\") pod \"cd01799a-fe39-4b3c-a57f-612e7c897c33\" (UID: \"cd01799a-fe39-4b3c-a57f-612e7c897c33\") " Oct 31 15:44:24.530668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.530642 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-openshift-ca" pod "cd01799a-fe39-4b3c-a57f-612e7c897c33" (UID: "cd01799a-fe39-4b3c-a57f-612e7c897c33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:24.531302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.531274 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:24.532773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.532730 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-xks-ca" pod "cd01799a-fe39-4b3c-a57f-612e7c897c33" (UID: "cd01799a-fe39-4b3c-a57f-612e7c897c33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:44:24.533048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.533017 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd01799a-fe39-4b3c-a57f-612e7c897c33-kube-api-access-wsm7k" pod "cd01799a-fe39-4b3c-a57f-612e7c897c33" (UID: "cd01799a-fe39-4b3c-a57f-612e7c897c33"). InnerVolumeSpecName "kube-api-access-wsm7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:24.632404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.632342 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd01799a-fe39-4b3c-a57f-612e7c897c33-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:24.632404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.632375 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsm7k\" (UniqueName: \"kubernetes.io/projected/cd01799a-fe39-4b3c-a57f-612e7c897c33-kube-api-access-wsm7k\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:24.701493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:24.701414 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:25.152034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:25.151995 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh"] Oct 31 15:44:25.255038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:25.255002 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" Oct 31 15:44:25.255038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:25.255023 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v" event={"ID":"cd01799a-fe39-4b3c-a57f-612e7c897c33","Type":"ContainerDied","Data":"f4eeb5152acf5e688ff8afc840bd693ce9d4c80be6040db16f371c4d6efc8b15"} Oct 31 15:44:25.255316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:25.255069 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4eeb5152acf5e688ff8afc840bd693ce9d4c80be6040db16f371c4d6efc8b15" Oct 31 15:44:25.260592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:25.260551 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h"] Oct 31 15:44:25.498027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:25.498000 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:25.623428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:25.623398 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:26.477299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.477264 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk"] Oct 31 15:44:26.477970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.477954 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cd01799a-fe39-4b3c-a57f-612e7c897c33" containerName="revoke-keys" Oct 31 15:44:26.478027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.477971 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd01799a-fe39-4b3c-a57f-612e7c897c33" containerName="revoke-keys" Oct 31 15:44:26.478105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.478095 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd01799a-fe39-4b3c-a57f-612e7c897c33" containerName="revoke-keys" Oct 31 15:44:26.481733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.481715 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.558493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.554864 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" Oct 31 15:44:26.611833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.611800 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk"] Oct 31 15:44:26.659624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.659592 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgq7w\" (UniqueName: \"kubernetes.io/projected/97891b87-02cb-4007-beed-5903c97d4380-kube-api-access-lgq7w\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.659820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.659714 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97891b87-02cb-4007-beed-5903c97d4380-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.659820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.659764 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.659962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.659840 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97891b87-02cb-4007-beed-5903c97d4380-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.659962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.659896 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.660060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.660015 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97891b87-02cb-4007-beed-5903c97d4380-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.660115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.660062 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.660115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.660079 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.660259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.660198 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.760919 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgq7w\" (UniqueName: \"kubernetes.io/projected/97891b87-02cb-4007-beed-5903c97d4380-kube-api-access-lgq7w\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761019 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97891b87-02cb-4007-beed-5903c97d4380-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761072 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761101 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97891b87-02cb-4007-beed-5903c97d4380-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761420 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761555 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97891b87-02cb-4007-beed-5903c97d4380-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761617 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761643 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.761743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.761716 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.762605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.762425 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.762605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.762499 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.762785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.762645 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.762785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.762683 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97891b87-02cb-4007-beed-5903c97d4380-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.764036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.763992 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.765632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.765607 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97891b87-02cb-4007-beed-5903c97d4380-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.766950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.766926 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97891b87-02cb-4007-beed-5903c97d4380-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.818861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.818826 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgq7w\" (UniqueName: \"kubernetes.io/projected/97891b87-02cb-4007-beed-5903c97d4380-kube-api-access-lgq7w\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:26.819005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:26.818985 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97891b87-02cb-4007-beed-5903c97d4380-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-r24tk\" (UID: \"97891b87-02cb-4007-beed-5903c97d4380\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:27.096503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:27.096384 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:27.531948 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:44:27.531914 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97891b87_02cb_4007_beed_5903c97d4380.slice/crio-62697ce5967be613880cb084353a946b1e63950bbf0ad70b813ddb7bac8fd016 WatchSource:0}: Error finding container 62697ce5967be613880cb084353a946b1e63950bbf0ad70b813ddb7bac8fd016: Status 404 returned error can't find the container with id 62697ce5967be613880cb084353a946b1e63950bbf0ad70b813ddb7bac8fd016 Oct 31 15:44:27.532112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:27.532093 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk"] Oct 31 15:44:27.534305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:27.534271 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:44:27.534398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:27.534336 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:44:27.534398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:27.534365 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:44:28.288484 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:28.288434 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" event={"ID":"97891b87-02cb-4007-beed-5903c97d4380","Type":"ContainerStarted","Data":"6ea26d860ba26b1726d9aa202fba32af089d70ee62cfd858d6c263c9a653bb21"} Oct 31 15:44:28.288913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:28.288512 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" event={"ID":"97891b87-02cb-4007-beed-5903c97d4380","Type":"ContainerStarted","Data":"62697ce5967be613880cb084353a946b1e63950bbf0ad70b813ddb7bac8fd016"} Oct 31 15:44:28.385814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:28.385779 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:28.423514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:28.421538 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podStartSLOduration=2.4214630440000002 podStartE2EDuration="2.421463044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:44:28.409030561 +0000 UTC m=+3514.028581962" watchObservedRunningTime="2026-10-31 15:44:28.421463044 +0000 UTC m=+3514.041014454" Oct 31 15:44:29.114495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:29.106869 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:29.114495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:29.107003 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 31 15:44:29.114495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:29.107047 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 31 15:44:30.103499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.102976 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 31 15:44:30.103499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.103061 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 31 15:44:30.152387 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.152344 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" podUID="b7498c45-6d3a-44e0-a70e-92b62b937315" containerName="payload-processing" containerID="cri-o://7fbc74a4a2f1fc99965e9fc653e2cfeec3e69a2d8b929bb365edffb57820baf2" gracePeriod=25 Oct 31 15:44:30.207036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.207000 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:30.261180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.261146 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" podUID="3068771f-a256-41f8-b2f2-84c9738c8408" containerName="payload-pre-processing" containerID="cri-o://584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6" gracePeriod=25 Oct 31 15:44:30.332815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.332775 4865 generic.go:356] "Generic (PLEG): container finished" podID="b7498c45-6d3a-44e0-a70e-92b62b937315" containerID="7fbc74a4a2f1fc99965e9fc653e2cfeec3e69a2d8b929bb365edffb57820baf2" exitCode=0 Oct 31 15:44:30.333501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.333375 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" event={"ID":"b7498c45-6d3a-44e0-a70e-92b62b937315","Type":"ContainerDied","Data":"7fbc74a4a2f1fc99965e9fc653e2cfeec3e69a2d8b929bb365edffb57820baf2"} Oct 31 15:44:30.470417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.470382 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:44:30.533801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.533768 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7498c45-6d3a-44e0-a70e-92b62b937315-kube-api-access-gkc65\" (UniqueName: \"kubernetes.io/projected/b7498c45-6d3a-44e0-a70e-92b62b937315-kube-api-access-gkc65\") pod \"b7498c45-6d3a-44e0-a70e-92b62b937315\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " Oct 31 15:44:30.534037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.533997 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7498c45-6d3a-44e0-a70e-92b62b937315-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7498c45-6d3a-44e0-a70e-92b62b937315-plugins-config-volume\") pod \"b7498c45-6d3a-44e0-a70e-92b62b937315\" (UID: \"b7498c45-6d3a-44e0-a70e-92b62b937315\") " Oct 31 15:44:30.535203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.535174 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7498c45-6d3a-44e0-a70e-92b62b937315-plugins-config-volume" pod "b7498c45-6d3a-44e0-a70e-92b62b937315" (UID: "b7498c45-6d3a-44e0-a70e-92b62b937315"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:30.540742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.540701 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7498c45-6d3a-44e0-a70e-92b62b937315-kube-api-access-gkc65" pod "b7498c45-6d3a-44e0-a70e-92b62b937315" (UID: "b7498c45-6d3a-44e0-a70e-92b62b937315"). InnerVolumeSpecName "kube-api-access-gkc65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:30.578832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.578807 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:44:30.635772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.635689 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3068771f-a256-41f8-b2f2-84c9738c8408-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3068771f-a256-41f8-b2f2-84c9738c8408-plugins-config-volume\") pod \"3068771f-a256-41f8-b2f2-84c9738c8408\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " Oct 31 15:44:30.635921 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.635797 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3068771f-a256-41f8-b2f2-84c9738c8408-kube-api-access-4pfrp\" (UniqueName: \"kubernetes.io/projected/3068771f-a256-41f8-b2f2-84c9738c8408-kube-api-access-4pfrp\") pod \"3068771f-a256-41f8-b2f2-84c9738c8408\" (UID: \"3068771f-a256-41f8-b2f2-84c9738c8408\") " Oct 31 15:44:30.636404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.636383 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkc65\" (UniqueName: \"kubernetes.io/projected/b7498c45-6d3a-44e0-a70e-92b62b937315-kube-api-access-gkc65\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:30.636446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.636412 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7498c45-6d3a-44e0-a70e-92b62b937315-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:30.636851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.636830 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3068771f-a256-41f8-b2f2-84c9738c8408-plugins-config-volume" pod "3068771f-a256-41f8-b2f2-84c9738c8408" (UID: "3068771f-a256-41f8-b2f2-84c9738c8408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:30.639385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.639358 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3068771f-a256-41f8-b2f2-84c9738c8408-kube-api-access-4pfrp" pod "3068771f-a256-41f8-b2f2-84c9738c8408" (UID: "3068771f-a256-41f8-b2f2-84c9738c8408"). InnerVolumeSpecName "kube-api-access-4pfrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:30.738041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.738005 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pfrp\" (UniqueName: \"kubernetes.io/projected/3068771f-a256-41f8-b2f2-84c9738c8408-kube-api-access-4pfrp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:30.738041 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:30.738036 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3068771f-a256-41f8-b2f2-84c9738c8408-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:31.112497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.108707 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 31 15:44:31.112497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.108782 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 31 15:44:31.340919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.340889 4865 generic.go:356] "Generic (PLEG): container finished" podID="3068771f-a256-41f8-b2f2-84c9738c8408" containerID="584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6" exitCode=0 Oct 31 15:44:31.341096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.340934 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" Oct 31 15:44:31.341096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.340973 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" event={"ID":"3068771f-a256-41f8-b2f2-84c9738c8408","Type":"ContainerDied","Data":"584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6"} Oct 31 15:44:31.341096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.341031 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h" event={"ID":"3068771f-a256-41f8-b2f2-84c9738c8408","Type":"ContainerDied","Data":"9802bd8877770eb03bc13b64bd6e38363415cf17b3500a62ff4619e2511d61a2"} Oct 31 15:44:31.341096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.341061 4865 scope.go:122] "RemoveContainer" containerID="584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6" Oct 31 15:44:31.342790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.342775 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" Oct 31 15:44:31.342790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.342778 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh" event={"ID":"b7498c45-6d3a-44e0-a70e-92b62b937315","Type":"ContainerDied","Data":"637141c71491d8b9473eb8403bc3071f677327bb15f4f2923a897854dfd54de7"} Oct 31 15:44:31.353054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.353034 4865 scope.go:122] "RemoveContainer" containerID="584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6" Oct 31 15:44:31.353332 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:31.353305 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6\": container with ID starting with 584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6 not found: ID does not exist" containerID="584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6" Oct 31 15:44:31.353439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.353339 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6"} err="failed to get container status \"584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6\": rpc error: code = NotFound desc = could not find container \"584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6\": container with ID starting with 584b4fa722b2c0d44eb45f4f675136394688cd4bd0d05d03c30567fd44840dc6 not found: ID does not exist" Oct 31 15:44:31.353439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.353358 4865 scope.go:122] "RemoveContainer" containerID="7fbc74a4a2f1fc99965e9fc653e2cfeec3e69a2d8b929bb365edffb57820baf2" Oct 31 15:44:31.535037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.535005 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h"] Oct 31 15:44:31.586019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.585985 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5a0cef-7f59b66775-8r65h"] Oct 31 15:44:31.674797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.674716 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh"] Oct 31 15:44:31.677976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:31.677943 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5a0cef-6559ff6b7f-98bgh"] Oct 31 15:44:32.113495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:32.102922 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 31 15:44:32.113495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:32.102986 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 31 15:44:32.520746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:32.520708 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3068771f-a256-41f8-b2f2-84c9738c8408" path="/var/lib/kubelet/pods/3068771f-a256-41f8-b2f2-84c9738c8408/volumes" Oct 31 15:44:32.521284 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:32.521260 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7498c45-6d3a-44e0-a70e-92b62b937315" path="/var/lib/kubelet/pods/b7498c45-6d3a-44e0-a70e-92b62b937315/volumes" Oct 31 15:44:33.116496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:33.104926 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 31 15:44:33.116496 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:33.104994 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 31 15:44:33.359833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:33.359799 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f"] Oct 31 15:44:33.385490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:33.385396 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:33.511033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:33.510939 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:34.097692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:34.097658 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 31 15:44:34.097945 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:34.097715 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 31 15:44:35.097083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:35.097043 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body= Oct 31 15:44:35.097625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:35.097112 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" Oct 31 15:44:35.815898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:35.815857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:35.949570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:35.949535 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:37.137602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:37.137569 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:44:37.138143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:37.137654 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:44:37.246616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:37.246585 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:37.247076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:37.247047 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:37.247850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:37.247831 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" Oct 31 15:44:38.029795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.029759 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:38.361013 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.360912 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" podUID="40ce96e1-69de-4d52-ba58-9da649ff2532" containerName="payload-processing" containerID="cri-o://eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7" gracePeriod=25 Oct 31 15:44:38.660320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.660286 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:44:38.755002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.754968 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-routing-overlay-volume\") pod \"40ce96e1-69de-4d52-ba58-9da649ff2532\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " Oct 31 15:44:38.755219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.755031 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-provider-credentials\") pod \"40ce96e1-69de-4d52-ba58-9da649ff2532\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " Oct 31 15:44:38.755219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.755104 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-kube-api-access-p5vqp\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-kube-api-access-p5vqp\") pod \"40ce96e1-69de-4d52-ba58-9da649ff2532\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " Oct 31 15:44:38.755360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.755228 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-plugins-config-volume\") pod \"40ce96e1-69de-4d52-ba58-9da649ff2532\" (UID: \"40ce96e1-69de-4d52-ba58-9da649ff2532\") " Oct 31 15:44:38.755463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.755433 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-routing-overlay-volume" pod "40ce96e1-69de-4d52-ba58-9da649ff2532" (UID: "40ce96e1-69de-4d52-ba58-9da649ff2532"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:38.755879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.755851 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-plugins-config-volume" pod "40ce96e1-69de-4d52-ba58-9da649ff2532" (UID: "40ce96e1-69de-4d52-ba58-9da649ff2532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:38.756080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.756062 4865 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-routing-overlay-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:38.756162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.756104 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ce96e1-69de-4d52-ba58-9da649ff2532-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:38.758538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.758511 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-kube-api-access-p5vqp" pod "40ce96e1-69de-4d52-ba58-9da649ff2532" (UID: "40ce96e1-69de-4d52-ba58-9da649ff2532"). InnerVolumeSpecName "kube-api-access-p5vqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:38.758670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.758544 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-provider-credentials" pod "40ce96e1-69de-4d52-ba58-9da649ff2532" (UID: "40ce96e1-69de-4d52-ba58-9da649ff2532"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:38.857282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.857242 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vqp\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-kube-api-access-p5vqp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:38.857282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:38.857268 4865 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40ce96e1-69de-4d52-ba58-9da649ff2532-provider-credentials\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:39.410631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.410593 4865 generic.go:356] "Generic (PLEG): container finished" podID="40ce96e1-69de-4d52-ba58-9da649ff2532" containerID="eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7" exitCode=0 Oct 31 15:44:39.411197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.410642 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" Oct 31 15:44:39.411197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.410693 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" event={"ID":"40ce96e1-69de-4d52-ba58-9da649ff2532","Type":"ContainerDied","Data":"eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7"} Oct 31 15:44:39.411197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.410766 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f" event={"ID":"40ce96e1-69de-4d52-ba58-9da649ff2532","Type":"ContainerDied","Data":"9eeebb06cb3d83f5cf929f47861a7745289a664499222dad7449eed04176d401"} Oct 31 15:44:39.411197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.410801 4865 scope.go:122] "RemoveContainer" containerID="eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7" Oct 31 15:44:39.426587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.426561 4865 scope.go:122] "RemoveContainer" containerID="eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7" Oct 31 15:44:39.426926 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:39.426894 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7\": container with ID starting with eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7 not found: ID does not exist" containerID="eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7" Oct 31 15:44:39.427029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.426935 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7"} err="failed to get container status \"eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7\": rpc error: code = NotFound desc = could not find container \"eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7\": container with ID starting with eb840e19526ee871233eb30960dd80a7ab2f654a3898064dd170a724f2203ba7 not found: ID does not exist" Oct 31 15:44:39.578019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.577985 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f"] Oct 31 15:44:39.585245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:39.585220 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c7bd584d6-v4b7f"] Oct 31 15:44:40.497632 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:40.497605 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ce96e1-69de-4d52-ba58-9da649ff2532" path="/var/lib/kubelet/pods/40ce96e1-69de-4d52-ba58-9da649ff2532/volumes" Oct 31 15:44:41.717201 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:41.717155 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:41.959989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:41.959679 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:42.082859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.082782 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:42.202664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.202632 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:42.245037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245005 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5"] Oct 31 15:44:42.245688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245672 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b7498c45-6d3a-44e0-a70e-92b62b937315" containerName="payload-processing" Oct 31 15:44:42.245688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245687 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7498c45-6d3a-44e0-a70e-92b62b937315" containerName="payload-processing" Oct 31 15:44:42.245783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245734 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3068771f-a256-41f8-b2f2-84c9738c8408" containerName="payload-pre-processing" Oct 31 15:44:42.245783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245740 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3068771f-a256-41f8-b2f2-84c9738c8408" containerName="payload-pre-processing" Oct 31 15:44:42.245783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245750 4865 cpu_manager.go:414] "containerMap: removing container" podUID="40ce96e1-69de-4d52-ba58-9da649ff2532" containerName="payload-processing" Oct 31 15:44:42.245783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245754 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ce96e1-69de-4d52-ba58-9da649ff2532" containerName="payload-processing" Oct 31 15:44:42.245927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245843 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7498c45-6d3a-44e0-a70e-92b62b937315" containerName="payload-processing" Oct 31 15:44:42.245927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245859 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3068771f-a256-41f8-b2f2-84c9738c8408" containerName="payload-pre-processing" Oct 31 15:44:42.245927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.245866 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ce96e1-69de-4d52-ba58-9da649ff2532" containerName="payload-processing" Oct 31 15:44:42.250737 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.250715 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.305571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.305544 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5"] Oct 31 15:44:42.327336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.327309 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.327506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.327360 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-routing-overlay-volume\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.327585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.327529 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-plugins-config-volume\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.327658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.327645 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq845\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-kube-api-access-pq845\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.351258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.351191 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-27f39a2fc752\"" Oct 31 15:44:42.351540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.351508 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 15:44:42.351540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.351524 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 15:44:42.351540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.351515 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 15:44:42.351540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.351518 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-j8sq2\"" Oct 31 15:44:42.354822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.354805 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 15:44:42.428811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.428775 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq845\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-kube-api-access-pq845\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.428985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.428868 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.428985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.428918 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-routing-overlay-volume\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.429088 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:42.429015 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-27f39a2fc752: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:42.429088 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:42.429038 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:42.429088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.429042 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-plugins-config-volume\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.429259 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:42.429107 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials podName:9d529d2b-4381-4299-8dab-02378aad80fb nodeName:}" failed. No retries permitted until 2026-10-31 15:44:42.929086149 +0000 UTC m=+3528.548637544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials") pod "payload-processing-external-model-5b49fc77cc-cdrd5" (UID: "9d529d2b-4381-4299-8dab-02378aad80fb") : secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:42.429682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.429655 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-routing-overlay-volume\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.429833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.429813 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-plugins-config-volume\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.487112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.487078 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq845\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-kube-api-access-pq845\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.746896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.746843 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:42.947573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:42.947540 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:42.947763 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:42.947696 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-27f39a2fc752: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:42.947763 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:42.947716 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:42.947867 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:42.947782 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials podName:9d529d2b-4381-4299-8dab-02378aad80fb nodeName:}" failed. No retries permitted until 2026-10-31 15:44:43.947768833 +0000 UTC m=+3529.567320221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials") pod "payload-processing-external-model-5b49fc77cc-cdrd5" (UID: "9d529d2b-4381-4299-8dab-02378aad80fb") : secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:43.114167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.114077 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5"] Oct 31 15:44:43.114417 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:43.114398 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" podUID="9d529d2b-4381-4299-8dab-02378aad80fb" Oct 31 15:44:43.282398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.282357 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:43.439591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.439515 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:43.448049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.448027 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:43.555773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.555747 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-kube-api-access-pq845\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-kube-api-access-pq845\") pod \"9d529d2b-4381-4299-8dab-02378aad80fb\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " Oct 31 15:44:43.555954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.555804 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-routing-overlay-volume\") pod \"9d529d2b-4381-4299-8dab-02378aad80fb\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " Oct 31 15:44:43.555954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.555866 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-plugins-config-volume\") pod \"9d529d2b-4381-4299-8dab-02378aad80fb\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " Oct 31 15:44:43.556203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.556181 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-routing-overlay-volume" pod "9d529d2b-4381-4299-8dab-02378aad80fb" (UID: "9d529d2b-4381-4299-8dab-02378aad80fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:43.556272 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.556256 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-plugins-config-volume" pod "9d529d2b-4381-4299-8dab-02378aad80fb" (UID: "9d529d2b-4381-4299-8dab-02378aad80fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:43.556426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.556409 4865 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-routing-overlay-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:43.556508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.556430 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d529d2b-4381-4299-8dab-02378aad80fb-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:43.558200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.558183 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-kube-api-access-pq845" pod "9d529d2b-4381-4299-8dab-02378aad80fb" (UID: "9d529d2b-4381-4299-8dab-02378aad80fb"). InnerVolumeSpecName "kube-api-access-pq845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:43.657611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.657582 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq845\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-kube-api-access-pq845\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:43.965608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:43.965559 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials\") pod \"payload-processing-external-model-5b49fc77cc-cdrd5\" (UID: \"9d529d2b-4381-4299-8dab-02378aad80fb\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:43.966403 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:43.966261 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-27f39a2fc752: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:43.966592 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:43.966576 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:43.966781 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:43.966770 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials podName:9d529d2b-4381-4299-8dab-02378aad80fb nodeName:}" failed. No retries permitted until 2026-10-31 15:44:45.966747247 +0000 UTC m=+3531.586298659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials") pod "payload-processing-external-model-5b49fc77cc-cdrd5" (UID: "9d529d2b-4381-4299-8dab-02378aad80fb") : secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:44.446237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:44.446157 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5" Oct 31 15:44:44.678977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:44.678947 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5"] Oct 31 15:44:44.682232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:44.682205 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-cdrd5"] Oct 31 15:44:44.776550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:44.776522 4865 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d529d2b-4381-4299-8dab-02378aad80fb-provider-credentials\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:46.497315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:46.497278 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d529d2b-4381-4299-8dab-02378aad80fb" path="/var/lib/kubelet/pods/9d529d2b-4381-4299-8dab-02378aad80fb/volumes" Oct 31 15:44:47.418571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:47.418537 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:47.508566 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:47.508524 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:44:47.887297 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:47.887262 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:47.986334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:47.986304 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2"] Oct 31 15:44:48.093997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:48.093963 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l"] Oct 31 15:44:48.126691 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:48.126660 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:48.251364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:48.251329 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:48.378538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:48.378510 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:48.498674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:48.498646 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:51.613130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:51.613089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:51.901826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:51.901719 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:52.753756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.753726 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x"] Oct 31 15:44:52.758457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.758434 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.803859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.803819 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktn6\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-kube-api-access-vktn6\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.804092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.804049 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.804278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.804256 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-routing-overlay-volume\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.804623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.804388 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.854491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.826127 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 15:44:52.854491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.826183 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 15:44:52.855538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.855506 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 15:44:52.880510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.880422 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 15:44:52.880728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.880562 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-fgvpr\"" Oct 31 15:44:52.880728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.880614 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-27f39a2fc752\"" Oct 31 15:44:52.884420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.884388 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x"] Oct 31 15:44:52.905534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.905482 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.905755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.905634 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktn6\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-kube-api-access-vktn6\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.905755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.905673 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.905895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.905821 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-routing-overlay-volume\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.906968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.906711 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-routing-overlay-volume\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.907589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.907541 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.909779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.909751 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.963802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.963774 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktn6\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-kube-api-access-vktn6\") pod \"payload-processing-external-model-5b49fc77cc-9mq5x\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:52.987689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:52.987654 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" podUID="c97da477-d93f-45a3-b56c-0ce7be5cc00a" containerName="payload-processing" containerID="cri-o://2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615" gracePeriod=25 Oct 31 15:44:53.074011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.073981 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:53.095078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.095038 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" podUID="91685308-585b-4a27-9636-bb675ddda875" containerName="payload-pre-processing" containerID="cri-o://87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d" gracePeriod=25 Oct 31 15:44:53.279185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.279158 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:44:53.311733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.311703 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97da477-d93f-45a3-b56c-0ce7be5cc00a-kube-api-access-k47t9\" (UniqueName: \"kubernetes.io/projected/c97da477-d93f-45a3-b56c-0ce7be5cc00a-kube-api-access-k47t9\") pod \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " Oct 31 15:44:53.311899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.311885 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97da477-d93f-45a3-b56c-0ce7be5cc00a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97da477-d93f-45a3-b56c-0ce7be5cc00a-plugins-config-volume\") pod \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\" (UID: \"c97da477-d93f-45a3-b56c-0ce7be5cc00a\") " Oct 31 15:44:53.312454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.312423 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97da477-d93f-45a3-b56c-0ce7be5cc00a-plugins-config-volume" pod "c97da477-d93f-45a3-b56c-0ce7be5cc00a" (UID: "c97da477-d93f-45a3-b56c-0ce7be5cc00a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:53.316938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.316910 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97da477-d93f-45a3-b56c-0ce7be5cc00a-kube-api-access-k47t9" pod "c97da477-d93f-45a3-b56c-0ce7be5cc00a" (UID: "c97da477-d93f-45a3-b56c-0ce7be5cc00a"). InnerVolumeSpecName "kube-api-access-k47t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:53.360717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.360689 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:44:53.413592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.413554 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91685308-585b-4a27-9636-bb675ddda875-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91685308-585b-4a27-9636-bb675ddda875-plugins-config-volume\") pod \"91685308-585b-4a27-9636-bb675ddda875\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " Oct 31 15:44:53.413894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.413873 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91685308-585b-4a27-9636-bb675ddda875-kube-api-access-mrl7t\" (UniqueName: \"kubernetes.io/projected/91685308-585b-4a27-9636-bb675ddda875-kube-api-access-mrl7t\") pod \"91685308-585b-4a27-9636-bb675ddda875\" (UID: \"91685308-585b-4a27-9636-bb675ddda875\") " Oct 31 15:44:53.413894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.413879 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91685308-585b-4a27-9636-bb675ddda875-plugins-config-volume" pod "91685308-585b-4a27-9636-bb675ddda875" (UID: "91685308-585b-4a27-9636-bb675ddda875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:44:53.414677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.414649 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91685308-585b-4a27-9636-bb675ddda875-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:53.414797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.414679 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47t9\" (UniqueName: \"kubernetes.io/projected/c97da477-d93f-45a3-b56c-0ce7be5cc00a-kube-api-access-k47t9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:53.414797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.414696 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c97da477-d93f-45a3-b56c-0ce7be5cc00a-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:53.416196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.416174 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91685308-585b-4a27-9636-bb675ddda875-kube-api-access-mrl7t" pod "91685308-585b-4a27-9636-bb675ddda875" (UID: "91685308-585b-4a27-9636-bb675ddda875"). InnerVolumeSpecName "kube-api-access-mrl7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:44:53.453282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.453251 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x"] Oct 31 15:44:53.492688 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:44:53.492659 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a54a535_ee8c_49ae_990f_3ae8fcfb95bf.slice/crio-736667916c74d764611ba8a7d520019b21b5dd10fa5c1184f899180695f4e1ec WatchSource:0}: Error finding container 736667916c74d764611ba8a7d520019b21b5dd10fa5c1184f899180695f4e1ec: Status 404 returned error can't find the container with id 736667916c74d764611ba8a7d520019b21b5dd10fa5c1184f899180695f4e1ec Oct 31 15:44:53.514698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.514661 4865 generic.go:356] "Generic (PLEG): container finished" podID="91685308-585b-4a27-9636-bb675ddda875" containerID="87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d" exitCode=0 Oct 31 15:44:53.514910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.514888 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" event={"ID":"91685308-585b-4a27-9636-bb675ddda875","Type":"ContainerDied","Data":"87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d"} Oct 31 15:44:53.514999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.514946 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" event={"ID":"91685308-585b-4a27-9636-bb675ddda875","Type":"ContainerDied","Data":"91b701aa2cf83464bc977870abc2c9d4de0d60d3bb04f49fff4512f8ff0cdbe9"} Oct 31 15:44:53.514999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.514974 4865 scope.go:122] "RemoveContainer" containerID="87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d" Oct 31 15:44:53.515154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.515140 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l" Oct 31 15:44:53.518638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.518608 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrl7t\" (UniqueName: \"kubernetes.io/projected/91685308-585b-4a27-9636-bb675ddda875-kube-api-access-mrl7t\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:44:53.519065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.519036 4865 generic.go:356] "Generic (PLEG): container finished" podID="c97da477-d93f-45a3-b56c-0ce7be5cc00a" containerID="2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615" exitCode=0 Oct 31 15:44:53.519248 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.519223 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" event={"ID":"c97da477-d93f-45a3-b56c-0ce7be5cc00a","Type":"ContainerDied","Data":"2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615"} Oct 31 15:44:53.519422 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.519398 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" event={"ID":"c97da477-d93f-45a3-b56c-0ce7be5cc00a","Type":"ContainerDied","Data":"c220a2a119109c90c646dccb34880cf5d3efd1f4ff08708f11eb3bfee739dc3a"} Oct 31 15:44:53.519560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.519252 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2" Oct 31 15:44:53.521176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.521114 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" event={"ID":"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf","Type":"ContainerStarted","Data":"736667916c74d764611ba8a7d520019b21b5dd10fa5c1184f899180695f4e1ec"} Oct 31 15:44:53.539661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.539637 4865 scope.go:122] "RemoveContainer" containerID="87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d" Oct 31 15:44:53.542047 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:53.540718 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d\": container with ID starting with 87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d not found: ID does not exist" containerID="87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d" Oct 31 15:44:53.542047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.540761 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d"} err="failed to get container status \"87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d\": rpc error: code = NotFound desc = could not find container \"87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d\": container with ID starting with 87d26ba2f0aa97a0d2833c9fa821c2cdc3e3a591a27aaadf3295024e675b2d0d not found: ID does not exist" Oct 31 15:44:53.542047 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.540786 4865 scope.go:122] "RemoveContainer" containerID="2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615" Oct 31 15:44:53.554938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.554912 4865 scope.go:122] "RemoveContainer" containerID="2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615" Oct 31 15:44:53.555315 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:53.555279 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615\": container with ID starting with 2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615 not found: ID does not exist" containerID="2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615" Oct 31 15:44:53.555404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.555322 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615"} err="failed to get container status \"2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615\": rpc error: code = NotFound desc = could not find container \"2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615\": container with ID starting with 2d6e3230808fe14a0194f218369aa13c6309cec07705bad6f864db8a3a132615 not found: ID does not exist" Oct 31 15:44:53.686830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.686798 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l"] Oct 31 15:44:53.692115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.692081 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4d1877-6dcc6bdd87-pt59l"] Oct 31 15:44:53.801411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.801319 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:53.804673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.804653 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2"] Oct 31 15:44:53.810221 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.810198 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4d1877-5847f46c45-27td2"] Oct 31 15:44:53.965677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:53.965642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:54.498934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:54.498903 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91685308-585b-4a27-9636-bb675ddda875" path="/var/lib/kubelet/pods/91685308-585b-4a27-9636-bb675ddda875/volumes" Oct 31 15:44:54.499376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:54.499362 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97da477-d93f-45a3-b56c-0ce7be5cc00a" path="/var/lib/kubelet/pods/c97da477-d93f-45a3-b56c-0ce7be5cc00a/volumes" Oct 31 15:44:54.587694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:54.586599 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:54.593783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:54.592832 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" event={"ID":"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf","Type":"ContainerStarted","Data":"3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248"} Oct 31 15:44:54.710989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:54.710932 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:54.718174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:54.718129 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" podStartSLOduration=2.7181155500000003 podStartE2EDuration="2.71811555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:44:54.715093017 +0000 UTC m=+3540.334644428" watchObservedRunningTime="2026-10-31 15:44:54.71811555 +0000 UTC m=+3540.337666994" Oct 31 15:44:54.750971 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:54.750894 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-27f39a2fc752: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:54.750971 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:54.750920 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:54.750971 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:54.750965 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:44:55.250951763 +0000 UTC m=+3540.870503155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:55.202950 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:55.202875 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x"] Oct 31 15:44:55.257857 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:55.257824 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-27f39a2fc752: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:55.257857 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:55.257851 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:55.258063 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:55.257902 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:44:56.257889334 +0000 UTC m=+3541.877440723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:55.338421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:55.338388 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:55.611189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:55.611157 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:44:55.611371 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:55.611257 4865 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" secret="" err="secret \"payload-processing-external-model-dockercfg-fgvpr\" not found" Oct 31 15:44:55.765070 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:55.765028 4865 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 15:44:55.765294 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:55.765110 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:44:56.265088707 +0000 UTC m=+3541.884640101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : configmap "payload-processing-external-model-plugins" not found Oct 31 15:44:55.910030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:55.909918 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:56.039847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:56.039815 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:56.279905 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:56.279868 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-27f39a2fc752: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:56.279905 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:56.279886 4865 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 15:44:56.279905 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:56.279898 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:56.280521 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:56.279958 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:44:57.279943829 +0000 UTC m=+3542.899495218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : configmap "payload-processing-external-model-plugins" not found Oct 31 15:44:56.280521 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:56.279981 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:44:58.279969138 +0000 UTC m=+3543.899520528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:56.649585 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:56.649436 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:56.857939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:56.857910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:57.307581 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:57.307550 4865 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 15:44:57.307973 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:57.307625 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:44:59.30760924 +0000 UTC m=+3544.927160651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : configmap "payload-processing-external-model-plugins" not found Oct 31 15:44:57.504508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:57.503860 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:57.963086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:57.963046 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:57.981136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:57.981110 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5"] Oct 31 15:44:57.981323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:57.981299 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" containerID="cri-o://51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8" gracePeriod=30 Oct 31 15:44:58.338567 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:58.338487 4865 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-27f39a2fc752: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:58.338567 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:58.338509 4865 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x: secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:58.338567 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:58.338560 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:45:02.338546489 +0000 UTC m=+3547.958097878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : secret "agc-cred-27f39a2fc752" not found Oct 31 15:44:58.713235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:58.713211 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:59.040328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:59.040230 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:59.165098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:44:59.165064 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:44:59.364266 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:59.364225 4865 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 31 15:44:59.364921 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:44:59.364902 4865 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume podName:7a54a535-ee8c-49ae-990f-3ae8fcfb95bf nodeName:}" failed. No retries permitted until 2026-10-31 15:45:03.36487713 +0000 UTC m=+3548.984428538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume") pod "payload-processing-external-model-5b49fc77cc-9mq5x" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf") : configmap "payload-processing-external-model-plugins" not found Oct 31 15:45:00.171862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.171822 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h"] Oct 31 15:45:00.173627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.173594 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c97da477-d93f-45a3-b56c-0ce7be5cc00a" containerName="payload-processing" Oct 31 15:45:00.173838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.173817 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97da477-d93f-45a3-b56c-0ce7be5cc00a" containerName="payload-processing" Oct 31 15:45:00.174023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.174007 4865 cpu_manager.go:414] "containerMap: removing container" podUID="91685308-585b-4a27-9636-bb675ddda875" containerName="payload-pre-processing" Oct 31 15:45:00.174144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.174129 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91685308-585b-4a27-9636-bb675ddda875" containerName="payload-pre-processing" Oct 31 15:45:00.174457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.174438 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97da477-d93f-45a3-b56c-0ce7be5cc00a" containerName="payload-processing" Oct 31 15:45:00.174618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.174604 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="91685308-585b-4a27-9636-bb675ddda875" containerName="payload-pre-processing" Oct 31 15:45:00.180267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.180241 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4"] Oct 31 15:45:00.180393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.180370 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.185282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.185262 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b"] Oct 31 15:45:00.185395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.185376 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.190307 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.190284 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.216345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.216323 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2"] Oct 31 15:45:00.220549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.220525 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.221837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.221818 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j"] Oct 31 15:45:00.225834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.225793 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.226943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.226920 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s"] Oct 31 15:45:00.232640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.232619 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf"] Oct 31 15:45:00.232764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.232732 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.237502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.237484 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726"] Oct 31 15:45:00.237719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.237559 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.243404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.243372 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x"] Oct 31 15:45:00.243635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.243616 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.251843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.251821 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf"] Oct 31 15:45:00.252711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.252413 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.256252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.256227 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.264210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.264156 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:45:00.264210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.264156 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:45:00.264386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.264259 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:45:00.267623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.267600 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4"] Oct 31 15:45:00.272626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.272608 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6"] Oct 31 15:45:00.272741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.272723 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.277031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.277015 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h"] Oct 31 15:45:00.277116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.277044 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4"] Oct 31 15:45:00.277157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.277122 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.279449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.279431 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b"] Oct 31 15:45:00.282523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.282493 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j"] Oct 31 15:45:00.332694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.326687 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s"] Oct 31 15:45:00.332694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.329015 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf"] Oct 31 15:45:00.338758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.334521 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x"] Oct 31 15:45:00.341953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.339626 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4"] Oct 31 15:45:00.346755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.346734 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.377220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377190 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x699z\" (UniqueName: \"kubernetes.io/projected/be7ea082-ba12-4657-b084-bb96fa22de47-kube-api-access-x699z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377241 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7ea082-ba12-4657-b084-bb96fa22de47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377295 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377325 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d9a22-22da-4522-9836-303805eec788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377355 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30a8f9-bb03-46d3-b427-af682644a990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377379 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0920ebf-b4dc-4bf1-80d4-577453362c54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377418 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2247dcd-3c58-4437-81ef-b466a28710f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377500 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a8e6c-6ce3-48b1-9816-8c4842d06c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.377591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377540 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8v2\" (UniqueName: \"kubernetes.io/projected/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-kube-api-access-bw8v2\") pod \"maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.377889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377821 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58sm\" (UniqueName: \"kubernetes.io/projected/d2247dcd-3c58-4437-81ef-b466a28710f8-kube-api-access-j58sm\") pod \"maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.377889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377867 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csh9x\" (UniqueName: \"kubernetes.io/projected/7c30a8f9-bb03-46d3-b427-af682644a990-kube-api-access-csh9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.377959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377904 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh778\" (UniqueName: \"kubernetes.io/projected/911d9a22-22da-4522-9836-303805eec788-kube-api-access-rh778\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.377959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377931 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9b917-d10e-44f8-b111-95c375a4ebc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.378025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.377958 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686np\" (UniqueName: \"kubernetes.io/projected/210a8e6c-6ce3-48b1-9816-8c4842d06c64-kube-api-access-686np\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.378069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378022 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d79372-75e0-419c-bff8-763faea71ff1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.378069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378045 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/50d79372-75e0-419c-bff8-763faea71ff1-kube-api-access-q5ckk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.378246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378222 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed090bb8-6b32-442a-b0f1-1169604b20cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.378299 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378275 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.378355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378339 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9dcm\" (UniqueName: \"kubernetes.io/projected/d0920ebf-b4dc-4bf1-80d4-577453362c54-kube-api-access-j9dcm\") pod \"maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.378421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378403 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6ls\" (UniqueName: \"kubernetes.io/projected/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-kube-api-access-sj6ls\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.378537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378436 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrv4h\" (UniqueName: \"kubernetes.io/projected/ed090bb8-6b32-442a-b0f1-1169604b20cf-kube-api-access-lrv4h\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.378537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378502 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a69a0e-1cbb-4cf4-982b-fae90350b824-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.378648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378537 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjrv\" (UniqueName: \"kubernetes.io/projected/c0a69a0e-1cbb-4cf4-982b-fae90350b824-kube-api-access-mwjrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.378648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.378596 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrkv\" (UniqueName: \"kubernetes.io/projected/9cd9b917-d10e-44f8-b111-95c375a4ebc3-kube-api-access-krrkv\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.379804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.379783 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6"] Oct 31 15:45:00.389956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.389930 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf"] Oct 31 15:45:00.396862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.396197 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2"] Oct 31 15:45:00.399885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.399862 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726"] Oct 31 15:45:00.405267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.404773 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4"] Oct 31 15:45:00.440520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.440228 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4"] Oct 31 15:45:00.440520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.440277 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-05c970284caa\"" Oct 31 15:45:00.440520 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.440371 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2bvm4\"" Oct 31 15:45:00.480150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480112 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7ea082-ba12-4657-b084-bb96fa22de47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.480318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480159 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-routing-overlay-volume\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.480318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480195 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.480318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480217 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d9a22-22da-4522-9836-303805eec788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.480318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480237 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30a8f9-bb03-46d3-b427-af682644a990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.480318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480253 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0920ebf-b4dc-4bf1-80d4-577453362c54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.480594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480373 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2247dcd-3c58-4437-81ef-b466a28710f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.480594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480537 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a8e6c-6ce3-48b1-9816-8c4842d06c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.480594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480588 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8v2\" (UniqueName: \"kubernetes.io/projected/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-kube-api-access-bw8v2\") pod \"maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.480761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480699 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58sm\" (UniqueName: \"kubernetes.io/projected/d2247dcd-3c58-4437-81ef-b466a28710f8-kube-api-access-j58sm\") pod \"maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.480761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480736 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnrl\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-kube-api-access-7jnrl\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.480865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480778 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csh9x\" (UniqueName: \"kubernetes.io/projected/7c30a8f9-bb03-46d3-b427-af682644a990-kube-api-access-csh9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.480865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480817 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh778\" (UniqueName: \"kubernetes.io/projected/911d9a22-22da-4522-9836-303805eec788-kube-api-access-rh778\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.480865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480844 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9b917-d10e-44f8-b111-95c375a4ebc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.481029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480873 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686np\" (UniqueName: \"kubernetes.io/projected/210a8e6c-6ce3-48b1-9816-8c4842d06c64-kube-api-access-686np\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.481029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480953 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d79372-75e0-419c-bff8-763faea71ff1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.481029 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.480978 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/50d79372-75e0-419c-bff8-763faea71ff1-kube-api-access-q5ckk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.481212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481031 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed090bb8-6b32-442a-b0f1-1169604b20cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.481212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481064 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.481212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481083 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7ea082-ba12-4657-b084-bb96fa22de47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.481212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481149 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9dcm\" (UniqueName: \"kubernetes.io/projected/d0920ebf-b4dc-4bf1-80d4-577453362c54-kube-api-access-j9dcm\") pod \"maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.481212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481203 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481220 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d9a22-22da-4522-9836-303805eec788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481212 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6ls\" (UniqueName: \"kubernetes.io/projected/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-kube-api-access-sj6ls\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481278 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrv4h\" (UniqueName: \"kubernetes.io/projected/ed090bb8-6b32-442a-b0f1-1169604b20cf-kube-api-access-lrv4h\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481302 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a8e6c-6ce3-48b1-9816-8c4842d06c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481319 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2247dcd-3c58-4437-81ef-b466a28710f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481337 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a69a0e-1cbb-4cf4-982b-fae90350b824-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481383 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-plugins-config-volume\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481415 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjrv\" (UniqueName: \"kubernetes.io/projected/c0a69a0e-1cbb-4cf4-982b-fae90350b824-kube-api-access-mwjrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.481519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481208 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30a8f9-bb03-46d3-b427-af682644a990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.482030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481691 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-provider-credentials\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.482030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481741 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrkv\" (UniqueName: \"kubernetes.io/projected/9cd9b917-d10e-44f8-b111-95c375a4ebc3-kube-api-access-krrkv\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.482030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.481827 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x699z\" (UniqueName: \"kubernetes.io/projected/be7ea082-ba12-4657-b084-bb96fa22de47-kube-api-access-x699z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.482182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.482046 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9b917-d10e-44f8-b111-95c375a4ebc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.482182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.482049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d79372-75e0-419c-bff8-763faea71ff1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.482182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.482110 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.482328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.482295 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed090bb8-6b32-442a-b0f1-1169604b20cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.482385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.482370 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a69a0e-1cbb-4cf4-982b-fae90350b824-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.482559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.482541 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0920ebf-b4dc-4bf1-80d4-577453362c54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.540542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.540456 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh778\" (UniqueName: \"kubernetes.io/projected/911d9a22-22da-4522-9836-303805eec788-kube-api-access-rh778\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.540728 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.540604 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9dcm\" (UniqueName: \"kubernetes.io/projected/d0920ebf-b4dc-4bf1-80d4-577453362c54-kube-api-access-j9dcm\") pod \"maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.541290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.541261 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686np\" (UniqueName: \"kubernetes.io/projected/210a8e6c-6ce3-48b1-9816-8c4842d06c64-kube-api-access-686np\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.541730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.541710 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58sm\" (UniqueName: \"kubernetes.io/projected/d2247dcd-3c58-4437-81ef-b466a28710f8-kube-api-access-j58sm\") pod \"maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.542060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.542037 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrv4h\" (UniqueName: \"kubernetes.io/projected/ed090bb8-6b32-442a-b0f1-1169604b20cf-kube-api-access-lrv4h\") pod \"maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.542862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.542841 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6ls\" (UniqueName: \"kubernetes.io/projected/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-kube-api-access-sj6ls\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.544078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.544055 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8v2\" (UniqueName: \"kubernetes.io/projected/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-kube-api-access-bw8v2\") pod \"maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.549577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.549521 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x699z\" (UniqueName: \"kubernetes.io/projected/be7ea082-ba12-4657-b084-bb96fa22de47-kube-api-access-x699z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.549734 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.549574 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/50d79372-75e0-419c-bff8-763faea71ff1-kube-api-access-q5ckk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.550275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.550241 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csh9x\" (UniqueName: \"kubernetes.io/projected/7c30a8f9-bb03-46d3-b427-af682644a990-kube-api-access-csh9x\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.551191 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.551148 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjrv\" (UniqueName: \"kubernetes.io/projected/c0a69a0e-1cbb-4cf4-982b-fae90350b824-kube-api-access-mwjrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.551428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.551409 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrkv\" (UniqueName: \"kubernetes.io/projected/9cd9b917-d10e-44f8-b111-95c375a4ebc3-kube-api-access-krrkv\") pod \"maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.553810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.553788 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:00.562661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.562643 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:00.572445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.572423 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:00.584384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.581875 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:00.584384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.583038 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnrl\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-kube-api-access-7jnrl\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.584384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.583233 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-plugins-config-volume\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.584384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.583298 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-provider-credentials\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.584384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.583360 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-routing-overlay-volume\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.584384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.584108 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-routing-overlay-volume\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.586265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.586234 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-plugins-config-volume\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.587900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.587851 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-provider-credentials\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.590506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.590065 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:00.607236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.607210 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:00.621296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.621267 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:00.670006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.656130 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:00.681516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.679931 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnrl\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-kube-api-access-7jnrl\") pod \"payload-processing-external-model-7d79c654fd-zq7z4\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.688362 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.686426 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:00.806509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.806241 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:00.813559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.810584 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:00.835429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.822680 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:00.839220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.838999 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:00.976436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:00.976385 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j"] Oct 31 15:45:01.239001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.238973 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s"] Oct 31 15:45:01.296003 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.295962 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded090bb8_6b32_442a_b0f1_1169604b20cf.slice/crio-ad11382745bc6e64d34e1862b1df588204d2b4a92856d64964e2f9b539e7dcc7 WatchSource:0}: Error finding container ad11382745bc6e64d34e1862b1df588204d2b4a92856d64964e2f9b539e7dcc7: Status 404 returned error can't find the container with id ad11382745bc6e64d34e1862b1df588204d2b4a92856d64964e2f9b539e7dcc7 Oct 31 15:45:01.359279 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.359253 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210a8e6c_6ce3_48b1_9816_8c4842d06c64.slice/crio-f23c1e8cde2edc56f0a2c4966a4002fbe7faf34c3c844b34eb8f36325feac3e6 WatchSource:0}: Error finding container f23c1e8cde2edc56f0a2c4966a4002fbe7faf34c3c844b34eb8f36325feac3e6: Status 404 returned error can't find the container with id f23c1e8cde2edc56f0a2c4966a4002fbe7faf34c3c844b34eb8f36325feac3e6 Oct 31 15:45:01.359450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.359428 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf"] Oct 31 15:45:01.369175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.369153 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6"] Oct 31 15:45:01.371253 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.371225 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911d9a22_22da_4522_9836_303805eec788.slice/crio-77442c6508035efb5c27a6beca20cb9a1644527ebe4c11612b5a15643d4b8b56 WatchSource:0}: Error finding container 77442c6508035efb5c27a6beca20cb9a1644527ebe4c11612b5a15643d4b8b56: Status 404 returned error can't find the container with id 77442c6508035efb5c27a6beca20cb9a1644527ebe4c11612b5a15643d4b8b56 Oct 31 15:45:01.372875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.372754 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726"] Oct 31 15:45:01.373348 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.373324 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a69a0e_1cbb_4cf4_982b_fae90350b824.slice/crio-1ea38d6acb7e3126f698dd2a1d53de7942be7164c5b2822d36a154b36c0b806b WatchSource:0}: Error finding container 1ea38d6acb7e3126f698dd2a1d53de7942be7164c5b2822d36a154b36c0b806b: Status 404 returned error can't find the container with id 1ea38d6acb7e3126f698dd2a1d53de7942be7164c5b2822d36a154b36c0b806b Oct 31 15:45:01.374913 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.374881 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d79372_75e0_419c_bff8_763faea71ff1.slice/crio-2c836649ed79d2b00bf14dc548b102625b859e786c8eeddddbb40fc65a95db14 WatchSource:0}: Error finding container 2c836649ed79d2b00bf14dc548b102625b859e786c8eeddddbb40fc65a95db14: Status 404 returned error can't find the container with id 2c836649ed79d2b00bf14dc548b102625b859e786c8eeddddbb40fc65a95db14 Oct 31 15:45:01.376222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.376201 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4"] Oct 31 15:45:01.404963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.404937 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf"] Oct 31 15:45:01.409679 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.409650 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2247dcd_3c58_4437_81ef_b466a28710f8.slice/crio-c1636b45ba048d7dcfbd33a9fcda69e7b29ecdbc8cac6f4c81b972e150c949dc WatchSource:0}: Error finding container c1636b45ba048d7dcfbd33a9fcda69e7b29ecdbc8cac6f4c81b972e150c949dc: Status 404 returned error can't find the container with id c1636b45ba048d7dcfbd33a9fcda69e7b29ecdbc8cac6f4c81b972e150c949dc Oct 31 15:45:01.440407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.440384 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4"] Oct 31 15:45:01.441720 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.441695 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9e4a04_9584_49c8_bc47_025770e59b9f.slice/crio-79c3e620cbb03f861adbd50490216b11e39cf1ff7910a65953d0e100f70e0234 WatchSource:0}: Error finding container 79c3e620cbb03f861adbd50490216b11e39cf1ff7910a65953d0e100f70e0234: Status 404 returned error can't find the container with id 79c3e620cbb03f861adbd50490216b11e39cf1ff7910a65953d0e100f70e0234 Oct 31 15:45:01.449333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.449310 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x"] Oct 31 15:45:01.452819 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.452792 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd9b917_d10e_44f8_b111_95c375a4ebc3.slice/crio-cea6d609317281de8c27432339d64c98475415c200c97658984d6edf9a527721 WatchSource:0}: Error finding container cea6d609317281de8c27432339d64c98475415c200c97658984d6edf9a527721: Status 404 returned error can't find the container with id cea6d609317281de8c27432339d64c98475415c200c97658984d6edf9a527721 Oct 31 15:45:01.523384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.523358 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h"] Oct 31 15:45:01.545181 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.545160 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b"] Oct 31 15:45:01.548245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.548225 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2"] Oct 31 15:45:01.603016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.602983 4865 patch_prober.go:29] interesting pod/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:45:01.603182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.603036 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:45:01.604360 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.604221 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b8b14d_0d38_4e87_9d75_68f087d7ec1f.slice/crio-001a8c3b375b4b0196294ea044dd9158e861b87d303a3bfc21e10cf0e671d930 WatchSource:0}: Error finding container 001a8c3b375b4b0196294ea044dd9158e861b87d303a3bfc21e10cf0e671d930: Status 404 returned error can't find the container with id 001a8c3b375b4b0196294ea044dd9158e861b87d303a3bfc21e10cf0e671d930 Oct 31 15:45:01.606042 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.606011 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0058f3f3_372b_4879_bb51_d6a61e4fa4c7.slice/crio-f54c9f7a048820dd791aed7eb53d579b7061309be23bd2d57f929e653938c644 WatchSource:0}: Error finding container f54c9f7a048820dd791aed7eb53d579b7061309be23bd2d57f929e653938c644: Status 404 returned error can't find the container with id f54c9f7a048820dd791aed7eb53d579b7061309be23bd2d57f929e653938c644 Oct 31 15:45:01.617968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.617938 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" podUID="7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" containerName="payload-processing" containerID="cri-o://3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248" gracePeriod=25 Oct 31 15:45:01.670726 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:01.670698 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0920ebf_b4dc_4bf1_80d4_577453362c54.slice/crio-14fc5a5794a78553d60fe9825696dfa22c9a0efc880725899fa42fd467cef0d5 WatchSource:0}: Error finding container 14fc5a5794a78553d60fe9825696dfa22c9a0efc880725899fa42fd467cef0d5: Status 404 returned error can't find the container with id 14fc5a5794a78553d60fe9825696dfa22c9a0efc880725899fa42fd467cef0d5 Oct 31 15:45:01.671850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.671824 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4"] Oct 31 15:45:01.719251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.719212 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" event={"ID":"2d9e4a04-9584-49c8-bc47-025770e59b9f","Type":"ContainerStarted","Data":"7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94"} Oct 31 15:45:01.719429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.719283 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:01.719429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.719307 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" event={"ID":"2d9e4a04-9584-49c8-bc47-025770e59b9f","Type":"ContainerStarted","Data":"79c3e620cbb03f861adbd50490216b11e39cf1ff7910a65953d0e100f70e0234"} Oct 31 15:45:01.720580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.720553 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerStarted","Data":"cea6d609317281de8c27432339d64c98475415c200c97658984d6edf9a527721"} Oct 31 15:45:01.721772 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.721747 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerStarted","Data":"c1636b45ba048d7dcfbd33a9fcda69e7b29ecdbc8cac6f4c81b972e150c949dc"} Oct 31 15:45:01.723048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.722931 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerStarted","Data":"f23c1e8cde2edc56f0a2c4966a4002fbe7faf34c3c844b34eb8f36325feac3e6"} Oct 31 15:45:01.724429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.724407 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" event={"ID":"50d79372-75e0-419c-bff8-763faea71ff1","Type":"ContainerStarted","Data":"2c836649ed79d2b00bf14dc548b102625b859e786c8eeddddbb40fc65a95db14"} Oct 31 15:45:01.725798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.725773 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" event={"ID":"911d9a22-22da-4522-9836-303805eec788","Type":"ContainerStarted","Data":"77442c6508035efb5c27a6beca20cb9a1644527ebe4c11612b5a15643d4b8b56"} Oct 31 15:45:01.727233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.727212 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" event={"ID":"be7ea082-ba12-4657-b084-bb96fa22de47","Type":"ContainerStarted","Data":"4e5f86cfee539e96933f656b82e794e05ccfdeda44e22a138606d61735b3ef1b"} Oct 31 15:45:01.728450 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.728428 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" event={"ID":"0058f3f3-372b-4879-bb51-d6a61e4fa4c7","Type":"ContainerStarted","Data":"f54c9f7a048820dd791aed7eb53d579b7061309be23bd2d57f929e653938c644"} Oct 31 15:45:01.729731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.729699 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" event={"ID":"53b8b14d-0d38-4e87-9d75-68f087d7ec1f","Type":"ContainerStarted","Data":"001a8c3b375b4b0196294ea044dd9158e861b87d303a3bfc21e10cf0e671d930"} Oct 31 15:45:01.730871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.730850 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerStarted","Data":"1ea38d6acb7e3126f698dd2a1d53de7942be7164c5b2822d36a154b36c0b806b"} Oct 31 15:45:01.732032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.732010 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" event={"ID":"ed090bb8-6b32-442a-b0f1-1169604b20cf","Type":"ContainerStarted","Data":"ad11382745bc6e64d34e1862b1df588204d2b4a92856d64964e2f9b539e7dcc7"} Oct 31 15:45:01.733348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.733329 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerStarted","Data":"faa5bd40334b06c0a7b2d7766505a575256f8390cf8e5392b9115124f55d7dea"} Oct 31 15:45:01.891793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:01.891730 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" podStartSLOduration=1.89171015 podStartE2EDuration="1.89171015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:01.887904127 +0000 UTC m=+3547.507455538" watchObservedRunningTime="2026-10-31 15:45:01.89171015 +0000 UTC m=+3547.511261562" Oct 31 15:45:02.258549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.258512 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:45:02.335606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.335577 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials\") pod \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " Oct 31 15:45:02.335776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.335674 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-routing-overlay-volume\") pod \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " Oct 31 15:45:02.335776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.335740 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume\") pod \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " Oct 31 15:45:02.335899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.335839 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-kube-api-access-vktn6\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-kube-api-access-vktn6\") pod \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\" (UID: \"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf\") " Oct 31 15:45:02.336051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.336020 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-routing-overlay-volume" pod "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:02.336170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.336068 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume" pod "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:02.336349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.336327 4865 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-routing-overlay-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:02.336349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.336350 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:02.338059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.338035 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials" pod "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:02.338148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.338063 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-kube-api-access-vktn6" pod "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" (UID: "7a54a535-ee8c-49ae-990f-3ae8fcfb95bf"). InnerVolumeSpecName "kube-api-access-vktn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:02.437537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.437511 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vktn6\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-kube-api-access-vktn6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:02.437537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.437539 4865 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf-provider-credentials\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:02.743800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.743771 4865 generic.go:356] "Generic (PLEG): container finished" podID="911d9a22-22da-4522-9836-303805eec788" containerID="cbda8eeb1057de28637379229ee519676ab20c74b71a70ceda5e8c5a9351757f" exitCode=60 Oct 31 15:45:02.743897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.743856 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" event={"ID":"911d9a22-22da-4522-9836-303805eec788","Type":"ContainerDied","Data":"cbda8eeb1057de28637379229ee519676ab20c74b71a70ceda5e8c5a9351757f"} Oct 31 15:45:02.744131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.744111 4865 scope.go:122] "RemoveContainer" containerID="cbda8eeb1057de28637379229ee519676ab20c74b71a70ceda5e8c5a9351757f" Oct 31 15:45:02.745733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.745706 4865 generic.go:356] "Generic (PLEG): container finished" podID="be7ea082-ba12-4657-b084-bb96fa22de47" containerID="064139240ec41fd39f2c9122c4f2a9b65b6622c1e29e0ffbc53acb04aff56006" exitCode=60 Oct 31 15:45:02.745846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.745808 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" event={"ID":"be7ea082-ba12-4657-b084-bb96fa22de47","Type":"ContainerDied","Data":"064139240ec41fd39f2c9122c4f2a9b65b6622c1e29e0ffbc53acb04aff56006"} Oct 31 15:45:02.746009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.745995 4865 scope.go:122] "RemoveContainer" containerID="064139240ec41fd39f2c9122c4f2a9b65b6622c1e29e0ffbc53acb04aff56006" Oct 31 15:45:02.749637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.749612 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" event={"ID":"d0920ebf-b4dc-4bf1-80d4-577453362c54","Type":"ContainerStarted","Data":"0d84644220b48daafe1e7999ee710981ddd78b75715bbcd0b824271b30a5f792"} Oct 31 15:45:02.749746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.749661 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" event={"ID":"d0920ebf-b4dc-4bf1-80d4-577453362c54","Type":"ContainerStarted","Data":"14fc5a5794a78553d60fe9825696dfa22c9a0efc880725899fa42fd467cef0d5"} Oct 31 15:45:02.751620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.751596 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" event={"ID":"53b8b14d-0d38-4e87-9d75-68f087d7ec1f","Type":"ContainerStarted","Data":"56111636c360c7d0e9c0a04e2b95914672ac39cba9d9d3ad64b6cc60811322f3"} Oct 31 15:45:02.753406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.753385 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerStarted","Data":"50d63f1a312b04702183e5d45be0829cd0718059b435b227ecfe13321b33f7a2"} Oct 31 15:45:02.755714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.755157 4865 generic.go:356] "Generic (PLEG): container finished" podID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerID="4f3345de9acb909fdb36466959a089120309f7f158ab16d3be8d41465c5ac637" exitCode=60 Oct 31 15:45:02.755714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.755254 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" event={"ID":"ed090bb8-6b32-442a-b0f1-1169604b20cf","Type":"ContainerDied","Data":"4f3345de9acb909fdb36466959a089120309f7f158ab16d3be8d41465c5ac637"} Oct 31 15:45:02.755714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.755411 4865 scope.go:122] "RemoveContainer" containerID="4f3345de9acb909fdb36466959a089120309f7f158ab16d3be8d41465c5ac637" Oct 31 15:45:02.757088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.757063 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerStarted","Data":"0a4c57cf16dc6910d30fa9c139ff08722707f96a7fb6074b29e10199a058cd9d"} Oct 31 15:45:02.758947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.758914 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerStarted","Data":"c879ca0ef14e14618a63338ac929d6f7913692c82b751e31017d2a0b2386f950"} Oct 31 15:45:02.760314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.760290 4865 generic.go:356] "Generic (PLEG): container finished" podID="7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" containerID="3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248" exitCode=0 Oct 31 15:45:02.760418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.760319 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" Oct 31 15:45:02.760418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.760366 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" event={"ID":"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf","Type":"ContainerDied","Data":"3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248"} Oct 31 15:45:02.760418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.760412 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x" event={"ID":"7a54a535-ee8c-49ae-990f-3ae8fcfb95bf","Type":"ContainerDied","Data":"736667916c74d764611ba8a7d520019b21b5dd10fa5c1184f899180695f4e1ec"} Oct 31 15:45:02.760626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.760444 4865 scope.go:122] "RemoveContainer" containerID="3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248" Oct 31 15:45:02.762118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.762094 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerStarted","Data":"7682dcc0409e20e08ecd97d9d81a0b43ed12e62c3746e9f222a372b7182568bf"} Oct 31 15:45:02.763900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.763865 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerStarted","Data":"03230b50e86d3f3737946df0becff025aa08e3cfad0dcd9a441e8f881f118322"} Oct 31 15:45:02.765594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.765574 4865 generic.go:356] "Generic (PLEG): container finished" podID="50d79372-75e0-419c-bff8-763faea71ff1" containerID="e8dd5f34a708b72f209354eb6f27bca2812885ffd2b8fcc9028917b67201f8f1" exitCode=60 Oct 31 15:45:02.765699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.765685 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" event={"ID":"50d79372-75e0-419c-bff8-763faea71ff1","Type":"ContainerDied","Data":"e8dd5f34a708b72f209354eb6f27bca2812885ffd2b8fcc9028917b67201f8f1"} Oct 31 15:45:02.765878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.765862 4865 scope.go:122] "RemoveContainer" containerID="e8dd5f34a708b72f209354eb6f27bca2812885ffd2b8fcc9028917b67201f8f1" Oct 31 15:45:02.978348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:02.978294 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" podStartSLOduration=2.978280151 podStartE2EDuration="2.978280151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:02.97796774 +0000 UTC m=+3548.597519151" watchObservedRunningTime="2026-10-31 15:45:02.978280151 +0000 UTC m=+3548.597831563" Oct 31 15:45:03.094460 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.094410 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" podStartSLOduration=3.094396212 podStartE2EDuration="3.094396212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:03.091575052 +0000 UTC m=+3548.711126462" watchObservedRunningTime="2026-10-31 15:45:03.094396212 +0000 UTC m=+3548.713947678" Oct 31 15:45:03.183754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.183707 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:03.185457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.185430 4865 scope.go:122] "RemoveContainer" containerID="3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248" Oct 31 15:45:03.185924 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:03.185898 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248\": container with ID starting with 3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248 not found: ID does not exist" containerID="3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248" Oct 31 15:45:03.186068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.185937 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248"} err="failed to get container status \"3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248\": rpc error: code = NotFound desc = could not find container \"3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248\": container with ID starting with 3459a1d85cf957d53dd4933ba9b4f43f0169bc0eea7f8c812ec17b7311dc8248 not found: ID does not exist" Oct 31 15:45:03.220190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.220128 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" podStartSLOduration=3.220110335 podStartE2EDuration="3.220110335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:03.216908279 +0000 UTC m=+3548.836459696" watchObservedRunningTime="2026-10-31 15:45:03.220110335 +0000 UTC m=+3548.839661746" Oct 31 15:45:03.278762 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.278717 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4"] Oct 31 15:45:03.310593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.310565 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:03.437922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.437894 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:45:03.438260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.438212 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:03.558896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.558787 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-envoy\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.558896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.558842 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-data\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.558896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.558887 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-kube-api-access-xwr6v\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-kube-api-access-xwr6v\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.559197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.558930 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-socket\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.559197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.558950 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-token\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-token\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.559197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.558979 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-podinfo\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.559197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559030 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istiod-ca-cert\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.559197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559060 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-certs\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.559197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559113 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-data" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:45:03.559197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559172 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-socket" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:45:03.559671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559273 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-credential-socket\") pod \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\" (UID: \"2c868694-55bc-4cb9-83cb-03e07dfe35ba\") " Oct 31 15:45:03.559671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559405 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-certs" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:45:03.559671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559488 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istiod-ca-cert" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:03.559891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.559716 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-credential-socket" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:45:03.560988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.560679 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.560988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.560706 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.560988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.560720 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.560988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.560734 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.560988 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.560749 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.562210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.562186 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-podinfo" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:45:03.562393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.562371 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-kube-api-access-xwr6v" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "kube-api-access-xwr6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:03.562437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.562415 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-envoy" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:45:03.594713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.594683 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-token" pod "2c868694-55bc-4cb9-83cb-03e07dfe35ba" (UID: "2c868694-55bc-4cb9-83cb-03e07dfe35ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:03.661845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.661818 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.661845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.661843 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwr6v\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-kube-api-access-xwr6v\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.662000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.661855 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.662000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.661867 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c868694-55bc-4cb9-83cb-03e07dfe35ba-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:03.733526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.733448 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" podStartSLOduration=3.733434923 podStartE2EDuration="3.733434923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:03.732945083 +0000 UTC m=+3549.352496489" watchObservedRunningTime="2026-10-31 15:45:03.733434923 +0000 UTC m=+3549.352986333" Oct 31 15:45:03.775195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.775161 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" event={"ID":"911d9a22-22da-4522-9836-303805eec788","Type":"ContainerStarted","Data":"10831c017d8f4b21b9ef1df5e34dc59a1f0203ddbc841166e6258a81598d6406"} Oct 31 15:45:03.777070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.777045 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" event={"ID":"be7ea082-ba12-4657-b084-bb96fa22de47","Type":"ContainerStarted","Data":"6e2bc4b2690e321a9026b0be9de27467b0c3527c2516128fc14276b11466c3db"} Oct 31 15:45:03.778518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.778494 4865 generic.go:356] "Generic (PLEG): container finished" podID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerID="0d84644220b48daafe1e7999ee710981ddd78b75715bbcd0b824271b30a5f792" exitCode=60 Oct 31 15:45:03.778615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.778525 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" event={"ID":"d0920ebf-b4dc-4bf1-80d4-577453362c54","Type":"ContainerDied","Data":"0d84644220b48daafe1e7999ee710981ddd78b75715bbcd0b824271b30a5f792"} Oct 31 15:45:03.778788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.778773 4865 scope.go:122] "RemoveContainer" containerID="0d84644220b48daafe1e7999ee710981ddd78b75715bbcd0b824271b30a5f792" Oct 31 15:45:03.780142 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.780121 4865 generic.go:356] "Generic (PLEG): container finished" podID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerID="b431ede84978f1fe03681287c0431455a1e6e53678b7067174e44e9b8a4154c9" exitCode=60 Oct 31 15:45:03.780227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.780185 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" event={"ID":"0058f3f3-372b-4879-bb51-d6a61e4fa4c7","Type":"ContainerDied","Data":"b431ede84978f1fe03681287c0431455a1e6e53678b7067174e44e9b8a4154c9"} Oct 31 15:45:03.780388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.780369 4865 scope.go:122] "RemoveContainer" containerID="b431ede84978f1fe03681287c0431455a1e6e53678b7067174e44e9b8a4154c9" Oct 31 15:45:03.781890 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.781869 4865 generic.go:356] "Generic (PLEG): container finished" podID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerID="56111636c360c7d0e9c0a04e2b95914672ac39cba9d9d3ad64b6cc60811322f3" exitCode=60 Oct 31 15:45:03.781991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.781902 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" event={"ID":"53b8b14d-0d38-4e87-9d75-68f087d7ec1f","Type":"ContainerDied","Data":"56111636c360c7d0e9c0a04e2b95914672ac39cba9d9d3ad64b6cc60811322f3"} Oct 31 15:45:03.782533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.782151 4865 scope.go:122] "RemoveContainer" containerID="56111636c360c7d0e9c0a04e2b95914672ac39cba9d9d3ad64b6cc60811322f3" Oct 31 15:45:03.783597 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.783575 4865 generic.go:356] "Generic (PLEG): container finished" podID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerID="50d63f1a312b04702183e5d45be0829cd0718059b435b227ecfe13321b33f7a2" exitCode=60 Oct 31 15:45:03.783678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.783651 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerDied","Data":"50d63f1a312b04702183e5d45be0829cd0718059b435b227ecfe13321b33f7a2"} Oct 31 15:45:03.783873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.783857 4865 scope.go:122] "RemoveContainer" containerID="50d63f1a312b04702183e5d45be0829cd0718059b435b227ecfe13321b33f7a2" Oct 31 15:45:03.785633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.785609 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" event={"ID":"ed090bb8-6b32-442a-b0f1-1169604b20cf","Type":"ContainerStarted","Data":"1228ca51fb00cd15da07c487fa792c72ed348e9307efec043182c745f147b8bf"} Oct 31 15:45:03.787218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.787199 4865 generic.go:356] "Generic (PLEG): container finished" podID="7c30a8f9-bb03-46d3-b427-af682644a990" containerID="0a4c57cf16dc6910d30fa9c139ff08722707f96a7fb6074b29e10199a058cd9d" exitCode=60 Oct 31 15:45:03.787318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.787275 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerDied","Data":"0a4c57cf16dc6910d30fa9c139ff08722707f96a7fb6074b29e10199a058cd9d"} Oct 31 15:45:03.787428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.787412 4865 scope.go:122] "RemoveContainer" containerID="0a4c57cf16dc6910d30fa9c139ff08722707f96a7fb6074b29e10199a058cd9d" Oct 31 15:45:03.788977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.788958 4865 generic.go:356] "Generic (PLEG): container finished" podID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerID="c879ca0ef14e14618a63338ac929d6f7913692c82b751e31017d2a0b2386f950" exitCode=60 Oct 31 15:45:03.789084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.789044 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerDied","Data":"c879ca0ef14e14618a63338ac929d6f7913692c82b751e31017d2a0b2386f950"} Oct 31 15:45:03.789178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.789162 4865 scope.go:122] "RemoveContainer" containerID="c879ca0ef14e14618a63338ac929d6f7913692c82b751e31017d2a0b2386f950" Oct 31 15:45:03.792648 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.792630 4865 generic.go:356] "Generic (PLEG): container finished" podID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerID="51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8" exitCode=0 Oct 31 15:45:03.792724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.792683 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" Oct 31 15:45:03.792724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.792697 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" event={"ID":"2c868694-55bc-4cb9-83cb-03e07dfe35ba","Type":"ContainerDied","Data":"51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8"} Oct 31 15:45:03.792854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.792737 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5" event={"ID":"2c868694-55bc-4cb9-83cb-03e07dfe35ba","Type":"ContainerDied","Data":"0784214c097bc81572f85fae3a22eb86761fcb888e1e63ee24dee693cbe2baa7"} Oct 31 15:45:03.792854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.792767 4865 scope.go:122] "RemoveContainer" containerID="51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8" Oct 31 15:45:03.796232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.796209 4865 generic.go:356] "Generic (PLEG): container finished" podID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerID="7682dcc0409e20e08ecd97d9d81a0b43ed12e62c3746e9f222a372b7182568bf" exitCode=60 Oct 31 15:45:03.796339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.796298 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerDied","Data":"7682dcc0409e20e08ecd97d9d81a0b43ed12e62c3746e9f222a372b7182568bf"} Oct 31 15:45:03.796650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.796632 4865 scope.go:122] "RemoveContainer" containerID="7682dcc0409e20e08ecd97d9d81a0b43ed12e62c3746e9f222a372b7182568bf" Oct 31 15:45:03.802961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.802940 4865 generic.go:356] "Generic (PLEG): container finished" podID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerID="03230b50e86d3f3737946df0becff025aa08e3cfad0dcd9a441e8f881f118322" exitCode=60 Oct 31 15:45:03.803064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.803038 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerDied","Data":"03230b50e86d3f3737946df0becff025aa08e3cfad0dcd9a441e8f881f118322"} Oct 31 15:45:03.803179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.803144 4865 scope.go:122] "RemoveContainer" containerID="03230b50e86d3f3737946df0becff025aa08e3cfad0dcd9a441e8f881f118322" Oct 31 15:45:03.805407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.805380 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" event={"ID":"50d79372-75e0-419c-bff8-763faea71ff1","Type":"ContainerStarted","Data":"fa45bb34b8f469054711a00c8e722e05bbe8dc74014bcd2d2bc39d3617cb45a5"} Oct 31 15:45:03.838033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.838009 4865 scope.go:122] "RemoveContainer" containerID="5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424" Oct 31 15:45:03.852888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.852831 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" podStartSLOduration=3.852812065 podStartE2EDuration="3.852812065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:03.85069625 +0000 UTC m=+3549.470247665" watchObservedRunningTime="2026-10-31 15:45:03.852812065 +0000 UTC m=+3549.472363478" Oct 31 15:45:03.861445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.861426 4865 scope.go:122] "RemoveContainer" containerID="51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8" Oct 31 15:45:03.862490 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:03.862335 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8\": container with ID starting with 51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8 not found: ID does not exist" containerID="51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8" Oct 31 15:45:03.862490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.862373 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8"} err="failed to get container status \"51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8\": rpc error: code = NotFound desc = could not find container \"51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8\": container with ID starting with 51e56388ffc494be1d34469ab2696135776736463f2ba7f05abcda9c93d937f8 not found: ID does not exist" Oct 31 15:45:03.862490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.862397 4865 scope.go:122] "RemoveContainer" containerID="5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424" Oct 31 15:45:03.892443 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:03.891532 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424\": container with ID starting with 5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424 not found: ID does not exist" containerID="5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424" Oct 31 15:45:03.892443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.891575 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424"} err="failed to get container status \"5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424\": rpc error: code = NotFound desc = could not find container \"5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424\": container with ID starting with 5f12c97640f34e899da161a161e2e2994b550b65b320742508c7adb4a1032424 not found: ID does not exist" Oct 31 15:45:03.967145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:03.967095 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" podStartSLOduration=3.967068733 podStartE2EDuration="3.967068733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:03.966140568 +0000 UTC m=+3549.585691978" watchObservedRunningTime="2026-10-31 15:45:03.967068733 +0000 UTC m=+3549.586620184" Oct 31 15:45:04.081908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.081794 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" podStartSLOduration=4.081780985 podStartE2EDuration="4.081780985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:04.080664019 +0000 UTC m=+3549.700215431" watchObservedRunningTime="2026-10-31 15:45:04.081780985 +0000 UTC m=+3549.701332395" Oct 31 15:45:04.265857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.265826 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5"] Oct 31 15:45:04.269312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.269245 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-982f43-openshift-default-66f447d649-659h5"] Oct 31 15:45:04.325160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.325130 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:04.502758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.502729 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" path="/var/lib/kubelet/pods/2c868694-55bc-4cb9-83cb-03e07dfe35ba/volumes" Oct 31 15:45:04.726072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.726021 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" podStartSLOduration=4.726007504 podStartE2EDuration="4.726007504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:04.721830086 +0000 UTC m=+3550.341381498" watchObservedRunningTime="2026-10-31 15:45:04.726007504 +0000 UTC m=+3550.345558957" Oct 31 15:45:04.815994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.815959 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerStarted","Data":"6be5c9156ddf51dc4cadfe4dca7a10a228a7dcd5c07187a4c8cd6997cd508330"} Oct 31 15:45:04.817804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.817779 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerStarted","Data":"79e4dfacdefe0a3b3f82e9ad089098fa739988b746e46608f4cb2288905bca27"} Oct 31 15:45:04.819595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.819572 4865 generic.go:356] "Generic (PLEG): container finished" podID="50d79372-75e0-419c-bff8-763faea71ff1" containerID="fa45bb34b8f469054711a00c8e722e05bbe8dc74014bcd2d2bc39d3617cb45a5" exitCode=60 Oct 31 15:45:04.819726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.819652 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" event={"ID":"50d79372-75e0-419c-bff8-763faea71ff1","Type":"ContainerDied","Data":"fa45bb34b8f469054711a00c8e722e05bbe8dc74014bcd2d2bc39d3617cb45a5"} Oct 31 15:45:04.819726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.819711 4865 scope.go:122] "RemoveContainer" containerID="e8dd5f34a708b72f209354eb6f27bca2812885ffd2b8fcc9028917b67201f8f1" Oct 31 15:45:04.819847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.819839 4865 scope.go:122] "RemoveContainer" containerID="fa45bb34b8f469054711a00c8e722e05bbe8dc74014bcd2d2bc39d3617cb45a5" Oct 31 15:45:04.820152 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:04.820124 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726_odh-ai-gateway-infra(50d79372-75e0-419c-bff8-763faea71ff1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" podUID="50d79372-75e0-419c-bff8-763faea71ff1" Oct 31 15:45:04.822403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.822383 4865 generic.go:356] "Generic (PLEG): container finished" podID="911d9a22-22da-4522-9836-303805eec788" containerID="10831c017d8f4b21b9ef1df5e34dc59a1f0203ddbc841166e6258a81598d6406" exitCode=60 Oct 31 15:45:04.822513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.822485 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" event={"ID":"911d9a22-22da-4522-9836-303805eec788","Type":"ContainerDied","Data":"10831c017d8f4b21b9ef1df5e34dc59a1f0203ddbc841166e6258a81598d6406"} Oct 31 15:45:04.822639 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.822626 4865 scope.go:122] "RemoveContainer" containerID="10831c017d8f4b21b9ef1df5e34dc59a1f0203ddbc841166e6258a81598d6406" Oct 31 15:45:04.822865 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:04.822847 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6_odh-ai-gateway-infra(911d9a22-22da-4522-9836-303805eec788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" podUID="911d9a22-22da-4522-9836-303805eec788" Oct 31 15:45:04.824269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.824247 4865 generic.go:356] "Generic (PLEG): container finished" podID="be7ea082-ba12-4657-b084-bb96fa22de47" containerID="6e2bc4b2690e321a9026b0be9de27467b0c3527c2516128fc14276b11466c3db" exitCode=60 Oct 31 15:45:04.824359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.824315 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" event={"ID":"be7ea082-ba12-4657-b084-bb96fa22de47","Type":"ContainerDied","Data":"6e2bc4b2690e321a9026b0be9de27467b0c3527c2516128fc14276b11466c3db"} Oct 31 15:45:04.824522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.824509 4865 scope.go:122] "RemoveContainer" containerID="6e2bc4b2690e321a9026b0be9de27467b0c3527c2516128fc14276b11466c3db" Oct 31 15:45:04.824769 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:04.824735 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j_odh-ai-gateway-infra(be7ea082-ba12-4657-b084-bb96fa22de47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" Oct 31 15:45:04.826228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.826209 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" event={"ID":"d0920ebf-b4dc-4bf1-80d4-577453362c54","Type":"ContainerStarted","Data":"170d1762d3b8ebbd2bdcc3e81796a43f5d513f6590e87f2626f16edc8503feca"} Oct 31 15:45:04.828263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.828233 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" event={"ID":"0058f3f3-372b-4879-bb51-d6a61e4fa4c7","Type":"ContainerStarted","Data":"1345695957e7d62af4501c871b1b067dad8d5c95212bab79e1224ab5a0390697"} Oct 31 15:45:04.830166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.830145 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" event={"ID":"53b8b14d-0d38-4e87-9d75-68f087d7ec1f","Type":"ContainerStarted","Data":"c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373"} Oct 31 15:45:04.832101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.832077 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerStarted","Data":"5d24d533a8764c3ff42ec247b4e993bdfdc4599320b2c5f26df7cd2e94405307"} Oct 31 15:45:04.836914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.834705 4865 generic.go:356] "Generic (PLEG): container finished" podID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerID="1228ca51fb00cd15da07c487fa792c72ed348e9307efec043182c745f147b8bf" exitCode=60 Oct 31 15:45:04.836914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.834812 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" event={"ID":"ed090bb8-6b32-442a-b0f1-1169604b20cf","Type":"ContainerDied","Data":"1228ca51fb00cd15da07c487fa792c72ed348e9307efec043182c745f147b8bf"} Oct 31 15:45:04.836914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.835185 4865 scope.go:122] "RemoveContainer" containerID="1228ca51fb00cd15da07c487fa792c72ed348e9307efec043182c745f147b8bf" Oct 31 15:45:04.836914 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:04.835481 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s_odh-ai-gateway-infra(ed090bb8-6b32-442a-b0f1-1169604b20cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" Oct 31 15:45:04.839833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.839809 4865 scope.go:122] "RemoveContainer" containerID="cbda8eeb1057de28637379229ee519676ab20c74b71a70ceda5e8c5a9351757f" Oct 31 15:45:04.841525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.841504 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerStarted","Data":"0739be0e7015b6fe85477c0e332bad31d092d14aae87112c6e843fbf20ce5b6e"} Oct 31 15:45:04.843401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.843382 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerStarted","Data":"97f1d608203b37780798c24c842a1abaf49d6849b532adeb229967e03af45516"} Oct 31 15:45:04.854431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.854413 4865 scope.go:122] "RemoveContainer" containerID="064139240ec41fd39f2c9122c4f2a9b65b6622c1e29e0ffbc53acb04aff56006" Oct 31 15:45:04.867778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:04.867761 4865 scope.go:122] "RemoveContainer" containerID="4f3345de9acb909fdb36466959a089120309f7f158ab16d3be8d41465c5ac637" Oct 31 15:45:05.182576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.182527 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" podStartSLOduration=5.18251236 podStartE2EDuration="5.18251236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:05.180244036 +0000 UTC m=+3550.799795449" watchObservedRunningTime="2026-10-31 15:45:05.18251236 +0000 UTC m=+3550.802063772" Oct 31 15:45:05.204610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.204580 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:05.214205 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.214182 4865 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b8b14d_0d38_4e87_9d75_68f087d7ec1f.slice/crio-conmon-c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373.scope\": RecentStats: unable to find data in memory cache]" Oct 31 15:45:05.301768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.301724 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" podStartSLOduration=5.301709772 podStartE2EDuration="5.301709772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:05.299028284 +0000 UTC m=+3550.918579694" watchObservedRunningTime="2026-10-31 15:45:05.301709772 +0000 UTC m=+3550.921261183" Oct 31 15:45:05.416825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.416724 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" podStartSLOduration=5.416710685 podStartE2EDuration="5.416710685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:05.413006216 +0000 UTC m=+3551.032557626" watchObservedRunningTime="2026-10-31 15:45:05.416710685 +0000 UTC m=+3551.036262136" Oct 31 15:45:05.423612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.423581 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:05.528059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.528027 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x"] Oct 31 15:45:05.531966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.531938 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b49fc77cc-9mq5x"] Oct 31 15:45:05.853084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.853052 4865 scope.go:122] "RemoveContainer" containerID="6e2bc4b2690e321a9026b0be9de27467b0c3527c2516128fc14276b11466c3db" Oct 31 15:45:05.853523 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.853461 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j_odh-ai-gateway-infra(be7ea082-ba12-4657-b084-bb96fa22de47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" Oct 31 15:45:05.855230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.855200 4865 generic.go:356] "Generic (PLEG): container finished" podID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerID="170d1762d3b8ebbd2bdcc3e81796a43f5d513f6590e87f2626f16edc8503feca" exitCode=60 Oct 31 15:45:05.855420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.855294 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" event={"ID":"d0920ebf-b4dc-4bf1-80d4-577453362c54","Type":"ContainerDied","Data":"170d1762d3b8ebbd2bdcc3e81796a43f5d513f6590e87f2626f16edc8503feca"} Oct 31 15:45:05.855420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.855346 4865 scope.go:122] "RemoveContainer" containerID="0d84644220b48daafe1e7999ee710981ddd78b75715bbcd0b824271b30a5f792" Oct 31 15:45:05.855629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.855606 4865 scope.go:122] "RemoveContainer" containerID="170d1762d3b8ebbd2bdcc3e81796a43f5d513f6590e87f2626f16edc8503feca" Oct 31 15:45:05.856044 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.855963 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4_odh-ai-gateway-infra(d0920ebf-b4dc-4bf1-80d4-577453362c54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" Oct 31 15:45:05.857909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.857878 4865 generic.go:356] "Generic (PLEG): container finished" podID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerID="1345695957e7d62af4501c871b1b067dad8d5c95212bab79e1224ab5a0390697" exitCode=60 Oct 31 15:45:05.858036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.857960 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" event={"ID":"0058f3f3-372b-4879-bb51-d6a61e4fa4c7","Type":"ContainerDied","Data":"1345695957e7d62af4501c871b1b067dad8d5c95212bab79e1224ab5a0390697"} Oct 31 15:45:05.858168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.858148 4865 scope.go:122] "RemoveContainer" containerID="1345695957e7d62af4501c871b1b067dad8d5c95212bab79e1224ab5a0390697" Oct 31 15:45:05.858411 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.858393 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2_odh-ai-gateway-infra(0058f3f3-372b-4879-bb51-d6a61e4fa4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" Oct 31 15:45:05.860036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.860017 4865 generic.go:356] "Generic (PLEG): container finished" podID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerID="c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373" exitCode=60 Oct 31 15:45:05.860144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.860083 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" event={"ID":"53b8b14d-0d38-4e87-9d75-68f087d7ec1f","Type":"ContainerDied","Data":"c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373"} Oct 31 15:45:05.860291 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.860276 4865 scope.go:122] "RemoveContainer" containerID="c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373" Oct 31 15:45:05.860549 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.860531 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b_odh-ai-gateway-infra(53b8b14d-0d38-4e87-9d75-68f087d7ec1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" Oct 31 15:45:05.862001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.861983 4865 generic.go:356] "Generic (PLEG): container finished" podID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerID="5d24d533a8764c3ff42ec247b4e993bdfdc4599320b2c5f26df7cd2e94405307" exitCode=60 Oct 31 15:45:05.862088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.862053 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerDied","Data":"5d24d533a8764c3ff42ec247b4e993bdfdc4599320b2c5f26df7cd2e94405307"} Oct 31 15:45:05.862208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.862192 4865 scope.go:122] "RemoveContainer" containerID="5d24d533a8764c3ff42ec247b4e993bdfdc4599320b2c5f26df7cd2e94405307" Oct 31 15:45:05.862421 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.862403 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4_odh-ai-gateway-infra(c0a69a0e-1cbb-4cf4-982b-fae90350b824)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" Oct 31 15:45:05.864005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.863976 4865 scope.go:122] "RemoveContainer" containerID="1228ca51fb00cd15da07c487fa792c72ed348e9307efec043182c745f147b8bf" Oct 31 15:45:05.864237 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.864217 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s_odh-ai-gateway-infra(ed090bb8-6b32-442a-b0f1-1169604b20cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" Oct 31 15:45:05.865633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.865612 4865 generic.go:356] "Generic (PLEG): container finished" podID="7c30a8f9-bb03-46d3-b427-af682644a990" containerID="0739be0e7015b6fe85477c0e332bad31d092d14aae87112c6e843fbf20ce5b6e" exitCode=60 Oct 31 15:45:05.865731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.865692 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerDied","Data":"0739be0e7015b6fe85477c0e332bad31d092d14aae87112c6e843fbf20ce5b6e"} Oct 31 15:45:05.865862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.865848 4865 scope.go:122] "RemoveContainer" containerID="0739be0e7015b6fe85477c0e332bad31d092d14aae87112c6e843fbf20ce5b6e" Oct 31 15:45:05.866068 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.866050 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h_odh-ai-gateway-infra(7c30a8f9-bb03-46d3-b427-af682644a990)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" Oct 31 15:45:05.867562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.867543 4865 generic.go:356] "Generic (PLEG): container finished" podID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerID="97f1d608203b37780798c24c842a1abaf49d6849b532adeb229967e03af45516" exitCode=60 Oct 31 15:45:05.867675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.867625 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerDied","Data":"97f1d608203b37780798c24c842a1abaf49d6849b532adeb229967e03af45516"} Oct 31 15:45:05.867806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.867791 4865 scope.go:122] "RemoveContainer" containerID="97f1d608203b37780798c24c842a1abaf49d6849b532adeb229967e03af45516" Oct 31 15:45:05.868036 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.868004 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x_odh-ai-gateway-infra(9cd9b917-d10e-44f8-b111-95c375a4ebc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" Oct 31 15:45:05.869680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.869659 4865 generic.go:356] "Generic (PLEG): container finished" podID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerID="6be5c9156ddf51dc4cadfe4dca7a10a228a7dcd5c07187a4c8cd6997cd508330" exitCode=60 Oct 31 15:45:05.869786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.869741 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerDied","Data":"6be5c9156ddf51dc4cadfe4dca7a10a228a7dcd5c07187a4c8cd6997cd508330"} Oct 31 15:45:05.869935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.869914 4865 scope.go:122] "RemoveContainer" containerID="6be5c9156ddf51dc4cadfe4dca7a10a228a7dcd5c07187a4c8cd6997cd508330" Oct 31 15:45:05.870128 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.870109 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf_odh-ai-gateway-infra(d2247dcd-3c58-4437-81ef-b466a28710f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" Oct 31 15:45:05.871186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.871170 4865 scope.go:122] "RemoveContainer" containerID="b431ede84978f1fe03681287c0431455a1e6e53678b7067174e44e9b8a4154c9" Oct 31 15:45:05.871870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.871848 4865 generic.go:356] "Generic (PLEG): container finished" podID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerID="79e4dfacdefe0a3b3f82e9ad089098fa739988b746e46608f4cb2288905bca27" exitCode=60 Oct 31 15:45:05.871957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.871924 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerDied","Data":"79e4dfacdefe0a3b3f82e9ad089098fa739988b746e46608f4cb2288905bca27"} Oct 31 15:45:05.872104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.872091 4865 scope.go:122] "RemoveContainer" containerID="79e4dfacdefe0a3b3f82e9ad089098fa739988b746e46608f4cb2288905bca27" Oct 31 15:45:05.872296 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.872274 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf_odh-ai-gateway-infra(210a8e6c-6ce3-48b1-9816-8c4842d06c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" Oct 31 15:45:05.873823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.873806 4865 scope.go:122] "RemoveContainer" containerID="fa45bb34b8f469054711a00c8e722e05bbe8dc74014bcd2d2bc39d3617cb45a5" Oct 31 15:45:05.874017 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.873999 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726_odh-ai-gateway-infra(50d79372-75e0-419c-bff8-763faea71ff1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" podUID="50d79372-75e0-419c-bff8-763faea71ff1" Oct 31 15:45:05.875329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.875313 4865 scope.go:122] "RemoveContainer" containerID="10831c017d8f4b21b9ef1df5e34dc59a1f0203ddbc841166e6258a81598d6406" Oct 31 15:45:05.875519 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:05.875499 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6_odh-ai-gateway-infra(911d9a22-22da-4522-9836-303805eec788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" podUID="911d9a22-22da-4522-9836-303805eec788" Oct 31 15:45:05.885672 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.885649 4865 scope.go:122] "RemoveContainer" containerID="56111636c360c7d0e9c0a04e2b95914672ac39cba9d9d3ad64b6cc60811322f3" Oct 31 15:45:05.906097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.906062 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2"] Oct 31 15:45:05.908924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.908903 4865 scope.go:122] "RemoveContainer" containerID="50d63f1a312b04702183e5d45be0829cd0718059b435b227ecfe13321b33f7a2" Oct 31 15:45:05.922609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.922587 4865 scope.go:122] "RemoveContainer" containerID="0a4c57cf16dc6910d30fa9c139ff08722707f96a7fb6074b29e10199a058cd9d" Oct 31 15:45:05.935694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.935673 4865 scope.go:122] "RemoveContainer" containerID="c879ca0ef14e14618a63338ac929d6f7913692c82b751e31017d2a0b2386f950" Oct 31 15:45:05.949839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.949816 4865 scope.go:122] "RemoveContainer" containerID="7682dcc0409e20e08ecd97d9d81a0b43ed12e62c3746e9f222a372b7182568bf" Oct 31 15:45:05.963045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:05.963025 4865 scope.go:122] "RemoveContainer" containerID="03230b50e86d3f3737946df0becff025aa08e3cfad0dcd9a441e8f881f118322" Oct 31 15:45:06.012854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.012823 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d"] Oct 31 15:45:06.250437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.250401 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:06.279565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.279536 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7585-b47afc"] Oct 31 15:45:06.280270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280255 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" Oct 31 15:45:06.280323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280273 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" Oct 31 15:45:06.280323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280298 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" containerName="payload-processing" Oct 31 15:45:06.280323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280303 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" containerName="payload-processing" Oct 31 15:45:06.280323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280319 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" Oct 31 15:45:06.280455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280324 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" Oct 31 15:45:06.280455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280429 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" containerName="payload-processing" Oct 31 15:45:06.280455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.280438 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" Oct 31 15:45:06.285319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.285300 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:06.357907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.355153 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6nthq\"" Oct 31 15:45:06.361711 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.361683 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-b47afc"] Oct 31 15:45:06.398770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.398734 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kg9\" (UniqueName: \"kubernetes.io/projected/e5ed88d3-b08d-4756-b844-193945de940c-kube-api-access-g2kg9\") pod \"test-curl-7585-b47afc\" (UID: \"e5ed88d3-b08d-4756-b844-193945de940c\") " pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:06.428141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.428092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:06.497959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.497926 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a54a535-ee8c-49ae-990f-3ae8fcfb95bf" path="/var/lib/kubelet/pods/7a54a535-ee8c-49ae-990f-3ae8fcfb95bf/volumes" Oct 31 15:45:06.499674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.499652 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kg9\" (UniqueName: \"kubernetes.io/projected/e5ed88d3-b08d-4756-b844-193945de940c-kube-api-access-g2kg9\") pod \"test-curl-7585-b47afc\" (UID: \"e5ed88d3-b08d-4756-b844-193945de940c\") " pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:06.557121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.557049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kg9\" (UniqueName: \"kubernetes.io/projected/e5ed88d3-b08d-4756-b844-193945de940c-kube-api-access-g2kg9\") pod \"test-curl-7585-b47afc\" (UID: \"e5ed88d3-b08d-4756-b844-193945de940c\") " pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:06.565806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.565780 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:06.598394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.598367 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:06.687498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.687317 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:06.884941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.884860 4865 scope.go:122] "RemoveContainer" containerID="97f1d608203b37780798c24c842a1abaf49d6849b532adeb229967e03af45516" Oct 31 15:45:06.885248 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.885224 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x_odh-ai-gateway-infra(9cd9b917-d10e-44f8-b111-95c375a4ebc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" Oct 31 15:45:06.886532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.886510 4865 scope.go:122] "RemoveContainer" containerID="6be5c9156ddf51dc4cadfe4dca7a10a228a7dcd5c07187a4c8cd6997cd508330" Oct 31 15:45:06.886825 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.886801 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf_odh-ai-gateway-infra(d2247dcd-3c58-4437-81ef-b466a28710f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" Oct 31 15:45:06.888147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.888130 4865 scope.go:122] "RemoveContainer" containerID="79e4dfacdefe0a3b3f82e9ad089098fa739988b746e46608f4cb2288905bca27" Oct 31 15:45:06.888438 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.888415 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf_odh-ai-gateway-infra(210a8e6c-6ce3-48b1-9816-8c4842d06c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" Oct 31 15:45:06.889754 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.889734 4865 scope.go:122] "RemoveContainer" containerID="170d1762d3b8ebbd2bdcc3e81796a43f5d513f6590e87f2626f16edc8503feca" Oct 31 15:45:06.890049 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.890029 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4_odh-ai-gateway-infra(d0920ebf-b4dc-4bf1-80d4-577453362c54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" Oct 31 15:45:06.891230 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.891211 4865 scope.go:122] "RemoveContainer" containerID="1345695957e7d62af4501c871b1b067dad8d5c95212bab79e1224ab5a0390697" Oct 31 15:45:06.891509 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.891485 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2_odh-ai-gateway-infra(0058f3f3-372b-4879-bb51-d6a61e4fa4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" Oct 31 15:45:06.892635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.892620 4865 scope.go:122] "RemoveContainer" containerID="c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373" Oct 31 15:45:06.892863 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.892839 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b_odh-ai-gateway-infra(53b8b14d-0d38-4e87-9d75-68f087d7ec1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" Oct 31 15:45:06.894074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.894054 4865 scope.go:122] "RemoveContainer" containerID="5d24d533a8764c3ff42ec247b4e993bdfdc4599320b2c5f26df7cd2e94405307" Oct 31 15:45:06.894283 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.894261 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4_odh-ai-gateway-infra(c0a69a0e-1cbb-4cf4-982b-fae90350b824)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" Oct 31 15:45:06.895593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.895577 4865 scope.go:122] "RemoveContainer" containerID="0739be0e7015b6fe85477c0e332bad31d092d14aae87112c6e843fbf20ce5b6e" Oct 31 15:45:06.895779 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:06.895762 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h_odh-ai-gateway-infra(7c30a8f9-bb03-46d3-b427-af682644a990)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" Oct 31 15:45:06.991605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.991578 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-b47afc"] Oct 31 15:45:06.992018 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:06.991999 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5ed88d3_b08d_4756_b844_193945de940c.slice/crio-fa5ec2b36ee8eb74ae89d7274ea6422fdf122297cba00dca620a73341d46da05 WatchSource:0}: Error finding container fa5ec2b36ee8eb74ae89d7274ea6422fdf122297cba00dca620a73341d46da05: Status 404 returned error can't find the container with id fa5ec2b36ee8eb74ae89d7274ea6422fdf122297cba00dca620a73341d46da05 Oct 31 15:45:06.994038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:06.994020 4865 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:45:07.191532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.191431 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj"] Oct 31 15:45:07.192298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.192283 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c868694-55bc-4cb9-83cb-03e07dfe35ba" containerName="istio-proxy" Oct 31 15:45:07.197357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.197336 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211243 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-envoy\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211276 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-token\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211307 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-data\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211398 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-certs\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211512 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvfl\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-kube-api-access-8qvfl\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211572 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a507b21c-dc8b-4741-a732-a5a3cb96f161-istiod-ca-cert\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211677 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-credential-socket\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211729 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-podinfo\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.211803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.211783 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-socket\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.253144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.253108 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj"] Oct 31 15:45:07.271109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.271082 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:07.290435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.290411 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b18df8-openshift-default-dockercfg-dlbw7\"" Oct 31 15:45:07.312809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.312767 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-envoy\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.312998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.312820 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-token\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.312998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.312871 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-data\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.312998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.312897 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-certs\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.312998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.312959 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvfl\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-kube-api-access-8qvfl\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313030 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a507b21c-dc8b-4741-a732-a5a3cb96f161-istiod-ca-cert\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313310 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-data\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313402 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313334 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-certs\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313423 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-credential-socket\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313507 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-podinfo\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313535 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-socket\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313736 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a507b21c-dc8b-4741-a732-a5a3cb96f161-istiod-ca-cert\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313794 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-credential-socket\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.313975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.313953 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-socket\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.315528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.315501 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-envoy\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.316144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.316118 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-podinfo\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.371253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.371222 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-token\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.371420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.371375 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvfl\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-kube-api-access-8qvfl\") pod \"e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.398094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.398063 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:07.513757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.513719 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:07.904011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.903929 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-b47afc" event={"ID":"e5ed88d3-b08d-4756-b844-193945de940c","Type":"ContainerStarted","Data":"fa5ec2b36ee8eb74ae89d7274ea6422fdf122297cba00dca620a73341d46da05"} Oct 31 15:45:07.926146 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:07.926097 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda507b21c_dc8b_4741_a732_a5a3cb96f161.slice/crio-c633be47406e3a4355578f86fc5899da337155d1baf70cdd81326351fb4f8010 WatchSource:0}: Error finding container c633be47406e3a4355578f86fc5899da337155d1baf70cdd81326351fb4f8010: Status 404 returned error can't find the container with id c633be47406e3a4355578f86fc5899da337155d1baf70cdd81326351fb4f8010 Oct 31 15:45:07.926763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.926736 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj"] Oct 31 15:45:07.929241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.929200 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:07.929323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.929287 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:07.929366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:07.929328 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:08.609526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.609499 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q"] Oct 31 15:45:08.613949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.613932 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.632971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.632908 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.633087 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.632955 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.633166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.633148 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82f2e0a-6663-44c5-be7f-3c22dbec5286-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.633225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.633207 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.633404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.633382 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6snq\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-kube-api-access-x6snq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.633523 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.633423 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.633599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.633546 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.667784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.667619 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 15:45:08.667784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.667751 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 15:45:08.667784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.667776 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 15:45:08.669462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.667834 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jwmrt\"" Oct 31 15:45:08.671936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.671913 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q"] Oct 31 15:45:08.734667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.734634 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.734863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.734742 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.734863 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.734775 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.734956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.734876 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82f2e0a-6663-44c5-be7f-3c22dbec5286-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.734956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.734935 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.735076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.735041 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6snq\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-kube-api-access-x6snq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.735147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.735075 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.735513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.735489 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.737611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.737587 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82f2e0a-6663-44c5-be7f-3c22dbec5286-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.737703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.737656 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.737798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.737776 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.737842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.737824 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.792810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.792735 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.792810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.792757 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6snq\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-kube-api-access-x6snq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8m82q\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:08.806561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.806529 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" podUID="2d9e4a04-9584-49c8-bc47-025770e59b9f" containerName="payload-processing" containerID="cri-o://7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94" gracePeriod=25 Oct 31 15:45:08.913657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.913617 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" event={"ID":"a507b21c-dc8b-4741-a732-a5a3cb96f161","Type":"ContainerStarted","Data":"eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723"} Oct 31 15:45:08.914088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.913672 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" event={"ID":"a507b21c-dc8b-4741-a732-a5a3cb96f161","Type":"ContainerStarted","Data":"c633be47406e3a4355578f86fc5899da337155d1baf70cdd81326351fb4f8010"} Oct 31 15:45:08.915424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.915391 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-b47afc" event={"ID":"e5ed88d3-b08d-4756-b844-193945de940c","Type":"ContainerStarted","Data":"385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a"} Oct 31 15:45:08.927045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:08.927020 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:09.043054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.042149 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" podStartSLOduration=2.042127753 podStartE2EDuration="2.042127753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:09.039760809 +0000 UTC m=+3554.659312223" watchObservedRunningTime="2026-10-31 15:45:09.042127753 +0000 UTC m=+3554.661679165" Oct 31 15:45:09.071640 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.071610 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:09.159217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.148309 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-plugins-config-volume\") pod \"2d9e4a04-9584-49c8-bc47-025770e59b9f\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " Oct 31 15:45:09.159217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.148362 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-kube-api-access-7jnrl\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-kube-api-access-7jnrl\") pod \"2d9e4a04-9584-49c8-bc47-025770e59b9f\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " Oct 31 15:45:09.159217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.148573 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-provider-credentials\") pod \"2d9e4a04-9584-49c8-bc47-025770e59b9f\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " Oct 31 15:45:09.159217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.148657 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-routing-overlay-volume\") pod \"2d9e4a04-9584-49c8-bc47-025770e59b9f\" (UID: \"2d9e4a04-9584-49c8-bc47-025770e59b9f\") " Oct 31 15:45:09.159217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.149687 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-routing-overlay-volume" pod "2d9e4a04-9584-49c8-bc47-025770e59b9f" (UID: "2d9e4a04-9584-49c8-bc47-025770e59b9f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:09.159217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.150071 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-plugins-config-volume" pod "2d9e4a04-9584-49c8-bc47-025770e59b9f" (UID: "2d9e4a04-9584-49c8-bc47-025770e59b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:09.159217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.156975 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-provider-credentials" pod "2d9e4a04-9584-49c8-bc47-025770e59b9f" (UID: "2d9e4a04-9584-49c8-bc47-025770e59b9f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:09.174459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.172816 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7585-b47afc" podStartSLOduration=1.655378397 podStartE2EDuration="3.172791939s" totalImagesPullingTime="1.517413542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:09.157650302 +0000 UTC m=+3554.777201714" watchObservedRunningTime="2026-10-31 15:45:09.172791939 +0000 UTC m=+3554.792343353" Oct 31 15:45:09.175300 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.175239 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-kube-api-access-7jnrl" pod "2d9e4a04-9584-49c8-bc47-025770e59b9f" (UID: "2d9e4a04-9584-49c8-bc47-025770e59b9f"). InnerVolumeSpecName "kube-api-access-7jnrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:09.250999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.250968 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:09.250999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.250995 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnrl\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-kube-api-access-7jnrl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:09.250999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.251006 4865 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9e4a04-9584-49c8-bc47-025770e59b9f-provider-credentials\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:09.251233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.251015 4865 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9e4a04-9584-49c8-bc47-025770e59b9f-routing-overlay-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:09.309797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.309391 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q"] Oct 31 15:45:09.310587 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:09.310539 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82f2e0a_6663_44c5_be7f_3c22dbec5286.slice/crio-86f35076e3eb73c34e92e37982eb72cc7a56e77c815dc9aa87abf9cee8c63537 WatchSource:0}: Error finding container 86f35076e3eb73c34e92e37982eb72cc7a56e77c815dc9aa87abf9cee8c63537: Status 404 returned error can't find the container with id 86f35076e3eb73c34e92e37982eb72cc7a56e77c815dc9aa87abf9cee8c63537 Oct 31 15:45:09.320258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.320215 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:09.320348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.320302 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:09.398395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.398366 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:09.514780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.514750 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:09.521060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.521033 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:09.923845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.923803 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" event={"ID":"f82f2e0a-6663-44c5-be7f-3c22dbec5286","Type":"ContainerStarted","Data":"237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd"} Oct 31 15:45:09.924334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.923861 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" event={"ID":"f82f2e0a-6663-44c5-be7f-3c22dbec5286","Type":"ContainerStarted","Data":"86f35076e3eb73c34e92e37982eb72cc7a56e77c815dc9aa87abf9cee8c63537"} Oct 31 15:45:09.924334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.923885 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:09.925211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.925186 4865 generic.go:356] "Generic (PLEG): container finished" podID="2d9e4a04-9584-49c8-bc47-025770e59b9f" containerID="7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94" exitCode=0 Oct 31 15:45:09.925321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.925228 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" Oct 31 15:45:09.925321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.925255 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" event={"ID":"2d9e4a04-9584-49c8-bc47-025770e59b9f","Type":"ContainerDied","Data":"7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94"} Oct 31 15:45:09.925321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.925291 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4" event={"ID":"2d9e4a04-9584-49c8-bc47-025770e59b9f","Type":"ContainerDied","Data":"79c3e620cbb03f861adbd50490216b11e39cf1ff7910a65953d0e100f70e0234"} Oct 31 15:45:09.925321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.925319 4865 scope.go:122] "RemoveContainer" containerID="7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94" Oct 31 15:45:09.925859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.925810 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:09.927185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.927165 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:45:09.935923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.935905 4865 scope.go:122] "RemoveContainer" containerID="7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94" Oct 31 15:45:09.936209 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:09.936187 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94\": container with ID starting with 7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94 not found: ID does not exist" containerID="7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94" Oct 31 15:45:09.936258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:09.936216 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94"} err="failed to get container status \"7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94\": rpc error: code = NotFound desc = could not find container \"7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94\": container with ID starting with 7c12ff6706e650ff2c34fff60142222ef94bc159232b76932533094b36927f94 not found: ID does not exist" Oct 31 15:45:10.054895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.054778 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" podStartSLOduration=2.054757528 podStartE2EDuration="2.054757528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:10.044373237 +0000 UTC m=+3555.663924650" watchObservedRunningTime="2026-10-31 15:45:10.054757528 +0000 UTC m=+3555.674308942" Oct 31 15:45:10.161745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.161712 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4"] Oct 31 15:45:10.165388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.165366 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d79c654fd-zq7z4"] Oct 31 15:45:10.500622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.500590 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9e4a04-9584-49c8-bc47-025770e59b9f" path="/var/lib/kubelet/pods/2d9e4a04-9584-49c8-bc47-025770e59b9f/volumes" Oct 31 15:45:10.602517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.580117 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7585-b47afc"] Oct 31 15:45:10.602517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.585961 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7585-b47afc"] Oct 31 15:45:10.906980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.906879 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" podUID="6a0480b5-63c2-4aca-9a4c-76741200a6a7" containerName="payload-processing" containerID="cri-o://c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a" gracePeriod=25 Oct 31 15:45:10.940789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.940755 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7585-b47afc" podUID="e5ed88d3-b08d-4756-b844-193945de940c" containerName="test-curl-7585-b47afc" containerID="cri-o://385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a" gracePeriod=2 Oct 31 15:45:10.945634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.945608 4865 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8m82q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:45:10.945749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:10.945657 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" podUID="f82f2e0a-6663-44c5-be7f-3c22dbec5286" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:45:10.992956 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:10.992920 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b47afc\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="e5ed88d3-b08d-4756-b844-193945de940c" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:11.016584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.016545 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" podUID="1da020dc-38f5-4282-8db7-3d88e9bb1f85" containerName="payload-pre-processing" containerID="cri-o://35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320" gracePeriod=25 Oct 31 15:45:11.230867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.230835 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:45:11.285002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.284954 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0480b5-63c2-4aca-9a4c-76741200a6a7-kube-api-access-gqhvn\" (UniqueName: \"kubernetes.io/projected/6a0480b5-63c2-4aca-9a4c-76741200a6a7-kube-api-access-gqhvn\") pod \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " Oct 31 15:45:11.285213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.285099 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0480b5-63c2-4aca-9a4c-76741200a6a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0480b5-63c2-4aca-9a4c-76741200a6a7-plugins-config-volume\") pod \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\" (UID: \"6a0480b5-63c2-4aca-9a4c-76741200a6a7\") " Oct 31 15:45:11.285544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.285516 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0480b5-63c2-4aca-9a4c-76741200a6a7-plugins-config-volume" pod "6a0480b5-63c2-4aca-9a4c-76741200a6a7" (UID: "6a0480b5-63c2-4aca-9a4c-76741200a6a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:11.285982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.285941 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0480b5-63c2-4aca-9a4c-76741200a6a7-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:11.298394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.298162 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0480b5-63c2-4aca-9a4c-76741200a6a7-kube-api-access-gqhvn" pod "6a0480b5-63c2-4aca-9a4c-76741200a6a7" (UID: "6a0480b5-63c2-4aca-9a4c-76741200a6a7"). InnerVolumeSpecName "kube-api-access-gqhvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:11.354575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.354457 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:45:11.387122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.387081 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da020dc-38f5-4282-8db7-3d88e9bb1f85-kube-api-access-bshwk\" (UniqueName: \"kubernetes.io/projected/1da020dc-38f5-4282-8db7-3d88e9bb1f85-kube-api-access-bshwk\") pod \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " Oct 31 15:45:11.387303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.387283 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da020dc-38f5-4282-8db7-3d88e9bb1f85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da020dc-38f5-4282-8db7-3d88e9bb1f85-plugins-config-volume\") pod \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\" (UID: \"1da020dc-38f5-4282-8db7-3d88e9bb1f85\") " Oct 31 15:45:11.387786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.387745 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da020dc-38f5-4282-8db7-3d88e9bb1f85-plugins-config-volume" pod "1da020dc-38f5-4282-8db7-3d88e9bb1f85" (UID: "1da020dc-38f5-4282-8db7-3d88e9bb1f85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:11.387965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.387945 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da020dc-38f5-4282-8db7-3d88e9bb1f85-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:11.388028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.387972 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqhvn\" (UniqueName: \"kubernetes.io/projected/6a0480b5-63c2-4aca-9a4c-76741200a6a7-kube-api-access-gqhvn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:11.390216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.390191 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da020dc-38f5-4282-8db7-3d88e9bb1f85-kube-api-access-bshwk" pod "1da020dc-38f5-4282-8db7-3d88e9bb1f85" (UID: "1da020dc-38f5-4282-8db7-3d88e9bb1f85"). InnerVolumeSpecName "kube-api-access-bshwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:11.398063 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:11.398033 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b47afc\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="e5ed88d3-b08d-4756-b844-193945de940c" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:11.450214 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:11.450178 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b47afc\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="e5ed88d3-b08d-4756-b844-193945de940c" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:11.489276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.489240 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshwk\" (UniqueName: \"kubernetes.io/projected/1da020dc-38f5-4282-8db7-3d88e9bb1f85-kube-api-access-bshwk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:11.723735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.723702 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:11.861700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.861608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:11.887875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.887840 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz"] Oct 31 15:45:11.888569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888551 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6a0480b5-63c2-4aca-9a4c-76741200a6a7" containerName="payload-processing" Oct 31 15:45:11.888569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888566 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0480b5-63c2-4aca-9a4c-76741200a6a7" containerName="payload-processing" Oct 31 15:45:11.888757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888617 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1da020dc-38f5-4282-8db7-3d88e9bb1f85" containerName="payload-pre-processing" Oct 31 15:45:11.888757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888623 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da020dc-38f5-4282-8db7-3d88e9bb1f85" containerName="payload-pre-processing" Oct 31 15:45:11.888757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888638 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e5ed88d3-b08d-4756-b844-193945de940c" containerName="test-curl-7585-b47afc" Oct 31 15:45:11.888757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888643 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ed88d3-b08d-4756-b844-193945de940c" containerName="test-curl-7585-b47afc" Oct 31 15:45:11.888757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888651 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2d9e4a04-9584-49c8-bc47-025770e59b9f" containerName="payload-processing" Oct 31 15:45:11.888757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888655 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9e4a04-9584-49c8-bc47-025770e59b9f" containerName="payload-processing" Oct 31 15:45:11.888757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888748 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0480b5-63c2-4aca-9a4c-76741200a6a7" containerName="payload-processing" Oct 31 15:45:11.889131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888764 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da020dc-38f5-4282-8db7-3d88e9bb1f85" containerName="payload-pre-processing" Oct 31 15:45:11.889131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888772 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ed88d3-b08d-4756-b844-193945de940c" containerName="test-curl-7585-b47afc" Oct 31 15:45:11.889131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.888778 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9e4a04-9584-49c8-bc47-025770e59b9f" containerName="payload-processing" Oct 31 15:45:11.892808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.892787 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.945014 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:11.944969 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b47afc\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="e5ed88d3-b08d-4756-b844-193945de940c" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:11.953616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.953583 4865 generic.go:356] "Generic (PLEG): container finished" podID="1da020dc-38f5-4282-8db7-3d88e9bb1f85" containerID="35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320" exitCode=0 Oct 31 15:45:11.953765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.953645 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" Oct 31 15:45:11.953765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.953637 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" event={"ID":"1da020dc-38f5-4282-8db7-3d88e9bb1f85","Type":"ContainerDied","Data":"35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320"} Oct 31 15:45:11.953907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.953806 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d" event={"ID":"1da020dc-38f5-4282-8db7-3d88e9bb1f85","Type":"ContainerDied","Data":"c317fc479951bd50185050a14880b438d1d62cce8b2cdde03f72f097407d56d0"} Oct 31 15:45:11.953907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.953867 4865 scope.go:122] "RemoveContainer" containerID="35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320" Oct 31 15:45:11.963561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.963375 4865 generic.go:356] "Generic (PLEG): container finished" podID="6a0480b5-63c2-4aca-9a4c-76741200a6a7" containerID="c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a" exitCode=0 Oct 31 15:45:11.963561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.963441 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" Oct 31 15:45:11.963561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.963483 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" event={"ID":"6a0480b5-63c2-4aca-9a4c-76741200a6a7","Type":"ContainerDied","Data":"c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a"} Oct 31 15:45:11.963561 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.963539 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2" event={"ID":"6a0480b5-63c2-4aca-9a4c-76741200a6a7","Type":"ContainerDied","Data":"d12b7011c5b01e43c82f7507508c587d371421bcfb68d843f7c5d6e7735a4bb5"} Oct 31 15:45:11.969313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.968991 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz"] Oct 31 15:45:11.986894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.986865 4865 scope.go:122] "RemoveContainer" containerID="35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320" Oct 31 15:45:11.987268 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:11.987246 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320\": container with ID starting with 35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320 not found: ID does not exist" containerID="35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320" Oct 31 15:45:11.987413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.987380 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320"} err="failed to get container status \"35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320\": rpc error: code = NotFound desc = could not find container \"35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320\": container with ID starting with 35535f93f81354622d35817265b7ede97945fa4197f44fb7a9bbdc93abe73320 not found: ID does not exist" Oct 31 15:45:11.987413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.987412 4865 scope.go:122] "RemoveContainer" containerID="c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a" Oct 31 15:45:11.995634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.995604 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.995725 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.995675 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.995846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.995814 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mkcj\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-kube-api-access-5mkcj\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.995965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.995945 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.996124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.996103 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.996211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.996165 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.996211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.996198 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.996344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.996320 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.996430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.996371 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:11.999763 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:11.999740 4865 scope.go:122] "RemoveContainer" containerID="c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a" Oct 31 15:45:12.000112 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:12.000087 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a\": container with ID starting with c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a not found: ID does not exist" containerID="c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a" Oct 31 15:45:12.000192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.000121 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a"} err="failed to get container status \"c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a\": rpc error: code = NotFound desc = could not find container \"c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a\": container with ID starting with c0420c616cf8c0db7932f4e5e8e189a43465485c0f6c5aca6149df451b0ffe4a not found: ID does not exist" Oct 31 15:45:12.010292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.010262 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:12.097555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.097527 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.097694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.097599 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.097750 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.097690 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.097817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.097753 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mkcj\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-kube-api-access-5mkcj\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.097960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.097912 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.097971 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098076 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098127 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098228 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098196 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098260 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098298 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098373 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098380 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.098994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.098929 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.100659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.100632 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.100756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.100663 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.113804 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:12.113732 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b47afc\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="e5ed88d3-b08d-4756-b844-193945de940c" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:12.157215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.157182 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mkcj\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-kube-api-access-5mkcj\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.157608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.157586 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-n69gz\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.207913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.207871 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:12.244198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.244164 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d"] Oct 31 15:45:12.262038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.262005 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-06ec34-6fd6df45ff-qpt8d"] Oct 31 15:45:12.373513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.373411 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2"] Oct 31 15:45:12.383967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.383934 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ec34-7d55475699-rvjt2"] Oct 31 15:45:12.497538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.497506 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da020dc-38f5-4282-8db7-3d88e9bb1f85" path="/var/lib/kubelet/pods/1da020dc-38f5-4282-8db7-3d88e9bb1f85/volumes" Oct 31 15:45:12.497969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.497956 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0480b5-63c2-4aca-9a4c-76741200a6a7" path="/var/lib/kubelet/pods/6a0480b5-63c2-4aca-9a4c-76741200a6a7/volumes" Oct 31 15:45:12.641816 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:12.641784 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7802fde4_f94f_49eb_be92_00fd1e5b5aec.slice/crio-f5bd3a823c8819d6f09965674ff01db62edad4f87eab369298e24c9b6eb3ee5b WatchSource:0}: Error finding container f5bd3a823c8819d6f09965674ff01db62edad4f87eab369298e24c9b6eb3ee5b: Status 404 returned error can't find the container with id f5bd3a823c8819d6f09965674ff01db62edad4f87eab369298e24c9b6eb3ee5b Oct 31 15:45:12.643408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.643382 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz"] Oct 31 15:45:12.644653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.644618 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:12.644756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.644685 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:12.644756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.644723 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:12.974498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.974438 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" event={"ID":"7802fde4-f94f-49eb-be92-00fd1e5b5aec","Type":"ContainerStarted","Data":"fbf18e8c199281845cecbaf21ee220c61e74dd3bfd4e49df433f5462d5d409df"} Oct 31 15:45:12.974902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:12.974520 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" event={"ID":"7802fde4-f94f-49eb-be92-00fd1e5b5aec","Type":"ContainerStarted","Data":"f5bd3a823c8819d6f09965674ff01db62edad4f87eab369298e24c9b6eb3ee5b"} Oct 31 15:45:13.096920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.096870 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podStartSLOduration=2.096854607 podStartE2EDuration="2.096854607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:13.094630141 +0000 UTC m=+3558.714181551" watchObservedRunningTime="2026-10-31 15:45:13.096854607 +0000 UTC m=+3558.716406015" Oct 31 15:45:13.208386 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.208362 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:13.210033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.210007 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:13.210156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.210054 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:13.225306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.225281 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:13.315882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.315851 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ed88d3-b08d-4756-b844-193945de940c-kube-api-access-g2kg9\" (UniqueName: \"kubernetes.io/projected/e5ed88d3-b08d-4756-b844-193945de940c-kube-api-access-g2kg9\") pod \"e5ed88d3-b08d-4756-b844-193945de940c\" (UID: \"e5ed88d3-b08d-4756-b844-193945de940c\") " Oct 31 15:45:13.318208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.318179 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ed88d3-b08d-4756-b844-193945de940c-kube-api-access-g2kg9" pod "e5ed88d3-b08d-4756-b844-193945de940c" (UID: "e5ed88d3-b08d-4756-b844-193945de940c"). InnerVolumeSpecName "kube-api-access-g2kg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:13.417976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.417940 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2kg9\" (UniqueName: \"kubernetes.io/projected/e5ed88d3-b08d-4756-b844-193945de940c-kube-api-access-g2kg9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:13.942220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.942191 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:45:13.986186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.986154 4865 generic.go:356] "Generic (PLEG): container finished" podID="e5ed88d3-b08d-4756-b844-193945de940c" containerID="385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a" exitCode=137 Oct 31 15:45:13.986677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.986207 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:13.986677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:13.986269 4865 scope.go:122] "RemoveContainer" containerID="385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a" Oct 31 15:45:14.022709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:14.022677 4865 scope.go:122] "RemoveContainer" containerID="385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a" Oct 31 15:45:14.023075 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:14.023047 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a\": container with ID starting with 385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a not found: ID does not exist" containerID="385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a" Oct 31 15:45:14.023202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:14.023085 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a"} err="failed to get container status \"385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a\": rpc error: code = NotFound desc = could not find container \"385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a\": container with ID starting with 385305ffc94b3404e267add6c5f423165c3bc396a1535a0d65e6beeac3aa7b3a not found: ID does not exist" Oct 31 15:45:14.174083 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:14.174040 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b47afc\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="e5ed88d3-b08d-4756-b844-193945de940c" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:14.221569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:14.217974 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:14.221569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:14.218038 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:14.521517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:14.521386 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ed88d3-b08d-4756-b844-193945de940c" path="/var/lib/kubelet/pods/e5ed88d3-b08d-4756-b844-193945de940c/volumes" Oct 31 15:45:14.545022 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:14.544982 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b47afc\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="e5ed88d3-b08d-4756-b844-193945de940c" pod="opendatahub/test-curl-7585-b47afc" Oct 31 15:45:14.784309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:14.784225 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:15.087124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:15.087009 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:15.219302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:15.210234 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:15.219302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:15.210305 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:15.273735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:15.273645 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:15.490044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:15.490009 4865 scope.go:122] "RemoveContainer" containerID="5d24d533a8764c3ff42ec247b4e993bdfdc4599320b2c5f26df7cd2e94405307" Oct 31 15:45:15.490213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:15.490154 4865 scope.go:122] "RemoveContainer" containerID="10831c017d8f4b21b9ef1df5e34dc59a1f0203ddbc841166e6258a81598d6406" Oct 31 15:45:16.035326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:16.035294 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerStarted","Data":"587dea25c14e7db2d2acc8198d86063d046ae38110423ccee48b7ebae8a199f2"} Oct 31 15:45:16.218850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:16.218813 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:16.219250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:16.218873 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:16.489784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:16.489758 4865 scope.go:122] "RemoveContainer" containerID="fa45bb34b8f469054711a00c8e722e05bbe8dc74014bcd2d2bc39d3617cb45a5" Oct 31 15:45:16.489956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:16.489923 4865 scope.go:122] "RemoveContainer" containerID="1345695957e7d62af4501c871b1b067dad8d5c95212bab79e1224ab5a0390697" Oct 31 15:45:16.490122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:16.490103 4865 scope.go:122] "RemoveContainer" containerID="c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373" Oct 31 15:45:16.490266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:16.490238 4865 scope.go:122] "RemoveContainer" containerID="79e4dfacdefe0a3b3f82e9ad089098fa739988b746e46608f4cb2288905bca27" Oct 31 15:45:17.063199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.063138 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerStarted","Data":"c682859d88f2bbf341197aab3946d2ec1264ad4b384f32b40292ca80819d264f"} Oct 31 15:45:17.065338 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.065309 4865 generic.go:356] "Generic (PLEG): container finished" podID="911d9a22-22da-4522-9836-303805eec788" containerID="9de792426b99f2608bce08d4393138f31bd915b28ff111166192429225f5df7b" exitCode=60 Oct 31 15:45:17.065516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.065348 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" event={"ID":"911d9a22-22da-4522-9836-303805eec788","Type":"ContainerDied","Data":"9de792426b99f2608bce08d4393138f31bd915b28ff111166192429225f5df7b"} Oct 31 15:45:17.065516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.065404 4865 scope.go:122] "RemoveContainer" containerID="10831c017d8f4b21b9ef1df5e34dc59a1f0203ddbc841166e6258a81598d6406" Oct 31 15:45:17.065865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.065839 4865 scope.go:122] "RemoveContainer" containerID="9de792426b99f2608bce08d4393138f31bd915b28ff111166192429225f5df7b" Oct 31 15:45:17.066220 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:17.066196 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6_odh-ai-gateway-infra(911d9a22-22da-4522-9836-303805eec788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" podUID="911d9a22-22da-4522-9836-303805eec788" Oct 31 15:45:17.067521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.067494 4865 generic.go:356] "Generic (PLEG): container finished" podID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerID="587dea25c14e7db2d2acc8198d86063d046ae38110423ccee48b7ebae8a199f2" exitCode=60 Oct 31 15:45:17.067660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.067642 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerDied","Data":"587dea25c14e7db2d2acc8198d86063d046ae38110423ccee48b7ebae8a199f2"} Oct 31 15:45:17.068032 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.068013 4865 scope.go:122] "RemoveContainer" containerID="587dea25c14e7db2d2acc8198d86063d046ae38110423ccee48b7ebae8a199f2" Oct 31 15:45:17.068342 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:17.068318 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4_odh-ai-gateway-infra(c0a69a0e-1cbb-4cf4-982b-fae90350b824)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" Oct 31 15:45:17.079608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.079591 4865 scope.go:122] "RemoveContainer" containerID="5d24d533a8764c3ff42ec247b4e993bdfdc4599320b2c5f26df7cd2e94405307" Oct 31 15:45:17.161120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.161096 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4"] Oct 31 15:45:17.215012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.214885 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:17.215012 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.214951 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:17.489075 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.489045 4865 scope.go:122] "RemoveContainer" containerID="6e2bc4b2690e321a9026b0be9de27467b0c3527c2516128fc14276b11466c3db" Oct 31 15:45:17.489565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.489215 4865 scope.go:122] "RemoveContainer" containerID="97f1d608203b37780798c24c842a1abaf49d6849b532adeb229967e03af45516" Oct 31 15:45:17.490925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:17.490910 4865 scope.go:122] "RemoveContainer" containerID="170d1762d3b8ebbd2bdcc3e81796a43f5d513f6590e87f2626f16edc8503feca" Oct 31 15:45:17.491373 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:17.491343 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:45:18.076996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.076961 4865 generic.go:356] "Generic (PLEG): container finished" podID="50d79372-75e0-419c-bff8-763faea71ff1" containerID="52c39998b8dd61dc5cbbb1b3ed19aa9ebdc03c2dc6b6a2d0ee4ace22bf5d53c2" exitCode=60 Oct 31 15:45:18.077170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.077028 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" event={"ID":"50d79372-75e0-419c-bff8-763faea71ff1","Type":"ContainerDied","Data":"52c39998b8dd61dc5cbbb1b3ed19aa9ebdc03c2dc6b6a2d0ee4ace22bf5d53c2"} Oct 31 15:45:18.077170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.077093 4865 scope.go:122] "RemoveContainer" containerID="fa45bb34b8f469054711a00c8e722e05bbe8dc74014bcd2d2bc39d3617cb45a5" Oct 31 15:45:18.077480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.077446 4865 scope.go:122] "RemoveContainer" containerID="52c39998b8dd61dc5cbbb1b3ed19aa9ebdc03c2dc6b6a2d0ee4ace22bf5d53c2" Oct 31 15:45:18.077839 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:18.077816 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726_odh-ai-gateway-infra(50d79372-75e0-419c-bff8-763faea71ff1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" podUID="50d79372-75e0-419c-bff8-763faea71ff1" Oct 31 15:45:18.080694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.080667 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" event={"ID":"be7ea082-ba12-4657-b084-bb96fa22de47","Type":"ContainerStarted","Data":"5cb8af0bbed38659867210692ca92220812ccc61bf83c77a54b153abb1ab451c"} Oct 31 15:45:18.082655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.082633 4865 generic.go:356] "Generic (PLEG): container finished" podID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerID="1e653085c06e3f34b90643ef951e0a42b9f61269638c5d4ae5b0a0c487cd8691" exitCode=60 Oct 31 15:45:18.082738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.082709 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" event={"ID":"0058f3f3-372b-4879-bb51-d6a61e4fa4c7","Type":"ContainerDied","Data":"1e653085c06e3f34b90643ef951e0a42b9f61269638c5d4ae5b0a0c487cd8691"} Oct 31 15:45:18.083080 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.083050 4865 scope.go:122] "RemoveContainer" containerID="1e653085c06e3f34b90643ef951e0a42b9f61269638c5d4ae5b0a0c487cd8691" Oct 31 15:45:18.083313 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:18.083292 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2_odh-ai-gateway-infra(0058f3f3-372b-4879-bb51-d6a61e4fa4c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" Oct 31 15:45:18.084689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.084671 4865 generic.go:356] "Generic (PLEG): container finished" podID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerID="0a1554f31a53d540d8d04b7d26e8fb0e3da8a00f480b9714f94cce3c37ca6efd" exitCode=60 Oct 31 15:45:18.084788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.084727 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" event={"ID":"53b8b14d-0d38-4e87-9d75-68f087d7ec1f","Type":"ContainerDied","Data":"0a1554f31a53d540d8d04b7d26e8fb0e3da8a00f480b9714f94cce3c37ca6efd"} Oct 31 15:45:18.085071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.085054 4865 scope.go:122] "RemoveContainer" containerID="0a1554f31a53d540d8d04b7d26e8fb0e3da8a00f480b9714f94cce3c37ca6efd" Oct 31 15:45:18.085327 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:18.085304 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b_odh-ai-gateway-infra(53b8b14d-0d38-4e87-9d75-68f087d7ec1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" Oct 31 15:45:18.088619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.088599 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerStarted","Data":"2f7e57bd05defdea291dba92e29c471e72c09cbb7a6988ae7db369b43140a414"} Oct 31 15:45:18.090927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.090910 4865 generic.go:356] "Generic (PLEG): container finished" podID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerID="c682859d88f2bbf341197aab3946d2ec1264ad4b384f32b40292ca80819d264f" exitCode=60 Oct 31 15:45:18.091002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.090935 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerDied","Data":"c682859d88f2bbf341197aab3946d2ec1264ad4b384f32b40292ca80819d264f"} Oct 31 15:45:18.091216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.091202 4865 scope.go:122] "RemoveContainer" containerID="c682859d88f2bbf341197aab3946d2ec1264ad4b384f32b40292ca80819d264f" Oct 31 15:45:18.091414 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:18.091398 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf_odh-ai-gateway-infra(210a8e6c-6ce3-48b1-9816-8c4842d06c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" Oct 31 15:45:18.092260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.092240 4865 scope.go:122] "RemoveContainer" containerID="1345695957e7d62af4501c871b1b067dad8d5c95212bab79e1224ab5a0390697" Oct 31 15:45:18.135631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.135597 4865 scope.go:122] "RemoveContainer" containerID="c3989042a3b94d9f1c18e882fccfedfb72ddc29ace1f0466d6608e7ff0197373" Oct 31 15:45:18.150810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.150788 4865 scope.go:122] "RemoveContainer" containerID="79e4dfacdefe0a3b3f82e9ad089098fa739988b746e46608f4cb2288905bca27" Oct 31 15:45:18.228657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.227857 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:18.228657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.227925 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:18.231396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.231359 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf"] Oct 31 15:45:18.283701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.283674 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:18.426212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.426184 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6"] Oct 31 15:45:18.445044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.445015 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a69a0e-1cbb-4cf4-982b-fae90350b824-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a69a0e-1cbb-4cf4-982b-fae90350b824-maas-api-service-ca\") pod \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " Oct 31 15:45:18.445175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.445086 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a69a0e-1cbb-4cf4-982b-fae90350b824-kube-api-access-mwjrv\" (UniqueName: \"kubernetes.io/projected/c0a69a0e-1cbb-4cf4-982b-fae90350b824-kube-api-access-mwjrv\") pod \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\" (UID: \"c0a69a0e-1cbb-4cf4-982b-fae90350b824\") " Oct 31 15:45:18.445431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.445405 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a69a0e-1cbb-4cf4-982b-fae90350b824-maas-api-service-ca" pod "c0a69a0e-1cbb-4cf4-982b-fae90350b824" (UID: "c0a69a0e-1cbb-4cf4-982b-fae90350b824"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:18.445811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.445768 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a69a0e-1cbb-4cf4-982b-fae90350b824-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:18.448113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.448083 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a69a0e-1cbb-4cf4-982b-fae90350b824-kube-api-access-mwjrv" pod "c0a69a0e-1cbb-4cf4-982b-fae90350b824" (UID: "c0a69a0e-1cbb-4cf4-982b-fae90350b824"). InnerVolumeSpecName "kube-api-access-mwjrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:18.489159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.489143 4865 scope.go:122] "RemoveContainer" containerID="0739be0e7015b6fe85477c0e332bad31d092d14aae87112c6e843fbf20ce5b6e" Oct 31 15:45:18.548112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.548089 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjrv\" (UniqueName: \"kubernetes.io/projected/c0a69a0e-1cbb-4cf4-982b-fae90350b824-kube-api-access-mwjrv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:18.879993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:18.879966 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:19.055243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.055209 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911d9a22-22da-4522-9836-303805eec788-kube-api-access-rh778\" (UniqueName: \"kubernetes.io/projected/911d9a22-22da-4522-9836-303805eec788-kube-api-access-rh778\") pod \"911d9a22-22da-4522-9836-303805eec788\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " Oct 31 15:45:19.055446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.055432 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911d9a22-22da-4522-9836-303805eec788-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d9a22-22da-4522-9836-303805eec788-maas-api-service-ca\") pod \"911d9a22-22da-4522-9836-303805eec788\" (UID: \"911d9a22-22da-4522-9836-303805eec788\") " Oct 31 15:45:19.055812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.055780 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911d9a22-22da-4522-9836-303805eec788-maas-api-service-ca" pod "911d9a22-22da-4522-9836-303805eec788" (UID: "911d9a22-22da-4522-9836-303805eec788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:19.056068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.056042 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d9a22-22da-4522-9836-303805eec788-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.057944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.057908 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911d9a22-22da-4522-9836-303805eec788-kube-api-access-rh778" pod "911d9a22-22da-4522-9836-303805eec788" (UID: "911d9a22-22da-4522-9836-303805eec788"). InnerVolumeSpecName "kube-api-access-rh778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:19.101677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.101643 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerStarted","Data":"81adc35469c4698b4df829edbd9a14b0c28420d949eb7286047815a3102894c3"} Oct 31 15:45:19.103738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.103707 4865 generic.go:356] "Generic (PLEG): container finished" podID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerID="2f7e57bd05defdea291dba92e29c471e72c09cbb7a6988ae7db369b43140a414" exitCode=60 Oct 31 15:45:19.103850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.103777 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerDied","Data":"2f7e57bd05defdea291dba92e29c471e72c09cbb7a6988ae7db369b43140a414"} Oct 31 15:45:19.103850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.103838 4865 scope.go:122] "RemoveContainer" containerID="97f1d608203b37780798c24c842a1abaf49d6849b532adeb229967e03af45516" Oct 31 15:45:19.104254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.104234 4865 scope.go:122] "RemoveContainer" containerID="2f7e57bd05defdea291dba92e29c471e72c09cbb7a6988ae7db369b43140a414" Oct 31 15:45:19.104640 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:19.104616 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x_odh-ai-gateway-infra(9cd9b917-d10e-44f8-b111-95c375a4ebc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" Oct 31 15:45:19.109628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.109606 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" Oct 31 15:45:19.109732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.109629 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6" event={"ID":"911d9a22-22da-4522-9836-303805eec788","Type":"ContainerDied","Data":"77442c6508035efb5c27a6beca20cb9a1644527ebe4c11612b5a15643d4b8b56"} Oct 31 15:45:19.111894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.111867 4865 generic.go:356] "Generic (PLEG): container finished" podID="be7ea082-ba12-4657-b084-bb96fa22de47" containerID="5cb8af0bbed38659867210692ca92220812ccc61bf83c77a54b153abb1ab451c" exitCode=60 Oct 31 15:45:19.112026 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.111948 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" event={"ID":"be7ea082-ba12-4657-b084-bb96fa22de47","Type":"ContainerDied","Data":"5cb8af0bbed38659867210692ca92220812ccc61bf83c77a54b153abb1ab451c"} Oct 31 15:45:19.112399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.112382 4865 scope.go:122] "RemoveContainer" containerID="5cb8af0bbed38659867210692ca92220812ccc61bf83c77a54b153abb1ab451c" Oct 31 15:45:19.112774 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:19.112751 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j_odh-ai-gateway-infra(be7ea082-ba12-4657-b084-bb96fa22de47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" Oct 31 15:45:19.125512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.122378 4865 generic.go:356] "Generic (PLEG): container finished" podID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerID="518a54655b3a43e9afefeaa73c5d8773b98d09013b70b478b2fa0ca0a00f77c7" exitCode=60 Oct 31 15:45:19.125512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.122497 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" event={"ID":"d0920ebf-b4dc-4bf1-80d4-577453362c54","Type":"ContainerDied","Data":"518a54655b3a43e9afefeaa73c5d8773b98d09013b70b478b2fa0ca0a00f77c7"} Oct 31 15:45:19.125512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.123042 4865 scope.go:122] "RemoveContainer" containerID="518a54655b3a43e9afefeaa73c5d8773b98d09013b70b478b2fa0ca0a00f77c7" Oct 31 15:45:19.125512 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:19.123609 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4_odh-ai-gateway-infra(d0920ebf-b4dc-4bf1-80d4-577453362c54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" Oct 31 15:45:19.131508 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.131449 4865 scope.go:122] "RemoveContainer" containerID="9de792426b99f2608bce08d4393138f31bd915b28ff111166192429225f5df7b" Oct 31 15:45:19.131780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.131756 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" event={"ID":"c0a69a0e-1cbb-4cf4-982b-fae90350b824","Type":"ContainerDied","Data":"1ea38d6acb7e3126f698dd2a1d53de7942be7164c5b2822d36a154b36c0b806b"} Oct 31 15:45:19.131889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.131857 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4" Oct 31 15:45:19.149815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.149752 4865 scope.go:122] "RemoveContainer" containerID="6e2bc4b2690e321a9026b0be9de27467b0c3527c2516128fc14276b11466c3db" Oct 31 15:45:19.159440 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.157535 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh778\" (UniqueName: \"kubernetes.io/projected/911d9a22-22da-4522-9836-303805eec788-kube-api-access-rh778\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.178436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.177916 4865 scope.go:122] "RemoveContainer" containerID="170d1762d3b8ebbd2bdcc3e81796a43f5d513f6590e87f2626f16edc8503feca" Oct 31 15:45:19.196578 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.196547 4865 scope.go:122] "RemoveContainer" containerID="587dea25c14e7db2d2acc8198d86063d046ae38110423ccee48b7ebae8a199f2" Oct 31 15:45:19.206217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.206195 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726"] Oct 31 15:45:19.209200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.209168 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:19.209306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.209240 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:19.341370 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.341339 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b"] Oct 31 15:45:19.365095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.365062 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:19.406570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.406541 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:19.449495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.449437 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j"] Oct 31 15:45:19.465519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.465463 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210a8e6c-6ce3-48b1-9816-8c4842d06c64-kube-api-access-686np\" (UniqueName: \"kubernetes.io/projected/210a8e6c-6ce3-48b1-9816-8c4842d06c64-kube-api-access-686np\") pod \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " Oct 31 15:45:19.465859 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.465776 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210a8e6c-6ce3-48b1-9816-8c4842d06c64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a8e6c-6ce3-48b1-9816-8c4842d06c64-maas-api-service-ca\") pod \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\" (UID: \"210a8e6c-6ce3-48b1-9816-8c4842d06c64\") " Oct 31 15:45:19.466705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.466676 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210a8e6c-6ce3-48b1-9816-8c4842d06c64-maas-api-service-ca" pod "210a8e6c-6ce3-48b1-9816-8c4842d06c64" (UID: "210a8e6c-6ce3-48b1-9816-8c4842d06c64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:19.475775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.475737 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210a8e6c-6ce3-48b1-9816-8c4842d06c64-kube-api-access-686np" pod "210a8e6c-6ce3-48b1-9816-8c4842d06c64" (UID: "210a8e6c-6ce3-48b1-9816-8c4842d06c64"). InnerVolumeSpecName "kube-api-access-686np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:19.489542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.489522 4865 scope.go:122] "RemoveContainer" containerID="1228ca51fb00cd15da07c487fa792c72ed348e9307efec043182c745f147b8bf" Oct 31 15:45:19.489932 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.489824 4865 scope.go:122] "RemoveContainer" containerID="6be5c9156ddf51dc4cadfe4dca7a10a228a7dcd5c07187a4c8cd6997cd508330" Oct 31 15:45:19.510591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.510565 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6"] Oct 31 15:45:19.514806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.514787 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891025-7h2m6"] Oct 31 15:45:19.534647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.534625 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:19.566693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.566665 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d79372-75e0-419c-bff8-763faea71ff1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d79372-75e0-419c-bff8-763faea71ff1-maas-api-service-ca\") pod \"50d79372-75e0-419c-bff8-763faea71ff1\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " Oct 31 15:45:19.566820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.566793 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d79372-75e0-419c-bff8-763faea71ff1-kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/50d79372-75e0-419c-bff8-763faea71ff1-kube-api-access-q5ckk\") pod \"50d79372-75e0-419c-bff8-763faea71ff1\" (UID: \"50d79372-75e0-419c-bff8-763faea71ff1\") " Oct 31 15:45:19.567045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.567017 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d79372-75e0-419c-bff8-763faea71ff1-maas-api-service-ca" pod "50d79372-75e0-419c-bff8-763faea71ff1" (UID: "50d79372-75e0-419c-bff8-763faea71ff1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:19.567423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.567405 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d79372-75e0-419c-bff8-763faea71ff1-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.567513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.567428 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a8e6c-6ce3-48b1-9816-8c4842d06c64-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.567513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.567443 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686np\" (UniqueName: \"kubernetes.io/projected/210a8e6c-6ce3-48b1-9816-8c4842d06c64-kube-api-access-686np\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.568941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.568920 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d79372-75e0-419c-bff8-763faea71ff1-kube-api-access-q5ckk" pod "50d79372-75e0-419c-bff8-763faea71ff1" (UID: "50d79372-75e0-419c-bff8-763faea71ff1"). InnerVolumeSpecName "kube-api-access-q5ckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:19.668404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.668371 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-kube-api-access-sj6ls\" (UniqueName: \"kubernetes.io/projected/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-kube-api-access-sj6ls\") pod \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " Oct 31 15:45:19.668733 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.668707 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-maas-api-service-ca\") pod \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\" (UID: \"53b8b14d-0d38-4e87-9d75-68f087d7ec1f\") " Oct 31 15:45:19.669656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.669625 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/50d79372-75e0-419c-bff8-763faea71ff1-kube-api-access-q5ckk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.669796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.669709 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-maas-api-service-ca" pod "53b8b14d-0d38-4e87-9d75-68f087d7ec1f" (UID: "53b8b14d-0d38-4e87-9d75-68f087d7ec1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:19.670906 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.670879 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-kube-api-access-sj6ls" pod "53b8b14d-0d38-4e87-9d75-68f087d7ec1f" (UID: "53b8b14d-0d38-4e87-9d75-68f087d7ec1f"). InnerVolumeSpecName "kube-api-access-sj6ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:19.680875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.680851 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2"] Oct 31 15:45:19.771692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.771660 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.771692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.771693 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj6ls\" (UniqueName: \"kubernetes.io/projected/53b8b14d-0d38-4e87-9d75-68f087d7ec1f-kube-api-access-sj6ls\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:19.911266 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.911235 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x"] Oct 31 15:45:19.919638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.919616 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:19.976213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.976173 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4"] Oct 31 15:45:19.979486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:19.979449 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-46bf36-29891025-xbkt4"] Oct 31 15:45:20.075528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.075497 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-maas-api-service-ca\") pod \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " Oct 31 15:45:20.075776 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.075756 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-kube-api-access-bw8v2\" (UniqueName: \"kubernetes.io/projected/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-kube-api-access-bw8v2\") pod \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\" (UID: \"0058f3f3-372b-4879-bb51-d6a61e4fa4c7\") " Oct 31 15:45:20.075913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.075892 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-maas-api-service-ca" pod "0058f3f3-372b-4879-bb51-d6a61e4fa4c7" (UID: "0058f3f3-372b-4879-bb51-d6a61e4fa4c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:20.076346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.076322 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.078121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.078091 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-kube-api-access-bw8v2" pod "0058f3f3-372b-4879-bb51-d6a61e4fa4c7" (UID: "0058f3f3-372b-4879-bb51-d6a61e4fa4c7"). InnerVolumeSpecName "kube-api-access-bw8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:20.117883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.117805 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:20.146163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.146131 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerStarted","Data":"a0a3b0b05c01821bf97c892bdfb19c43f27ed577a9b00a24e9d63a002ae5e23b"} Oct 31 15:45:20.147905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.147882 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" Oct 31 15:45:20.148025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.147900 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf" event={"ID":"210a8e6c-6ce3-48b1-9816-8c4842d06c64","Type":"ContainerDied","Data":"f23c1e8cde2edc56f0a2c4966a4002fbe7faf34c3c844b34eb8f36325feac3e6"} Oct 31 15:45:20.148025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.147965 4865 scope.go:122] "RemoveContainer" containerID="c682859d88f2bbf341197aab3946d2ec1264ad4b384f32b40292ca80819d264f" Oct 31 15:45:20.149861 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.149836 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" event={"ID":"50d79372-75e0-419c-bff8-763faea71ff1","Type":"ContainerDied","Data":"2c836649ed79d2b00bf14dc548b102625b859e786c8eeddddbb40fc65a95db14"} Oct 31 15:45:20.149965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.149929 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726" Oct 31 15:45:20.172777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.159818 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" event={"ID":"0058f3f3-372b-4879-bb51-d6a61e4fa4c7","Type":"ContainerDied","Data":"f54c9f7a048820dd791aed7eb53d579b7061309be23bd2d57f929e653938c644"} Oct 31 15:45:20.172777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.159838 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2" Oct 31 15:45:20.172777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.164334 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" event={"ID":"53b8b14d-0d38-4e87-9d75-68f087d7ec1f","Type":"ContainerDied","Data":"001a8c3b375b4b0196294ea044dd9158e861b87d303a3bfc21e10cf0e671d930"} Oct 31 15:45:20.172777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.164428 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b" Oct 31 15:45:20.173978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.173928 4865 scope.go:122] "RemoveContainer" containerID="52c39998b8dd61dc5cbbb1b3ed19aa9ebdc03c2dc6b6a2d0ee4ace22bf5d53c2" Oct 31 15:45:20.186730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.184568 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8v2\" (UniqueName: \"kubernetes.io/projected/0058f3f3-372b-4879-bb51-d6a61e4fa4c7-kube-api-access-bw8v2\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.198509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.197702 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" event={"ID":"ed090bb8-6b32-442a-b0f1-1169604b20cf","Type":"ContainerStarted","Data":"b1a3ce1c224f07779d677c472fe6abb56340627c36f7ef3ebc767063a45664f6"} Oct 31 15:45:20.237274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.209905 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:20.237274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.209966 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:20.237274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.211630 4865 generic.go:356] "Generic (PLEG): container finished" podID="7c30a8f9-bb03-46d3-b427-af682644a990" containerID="81adc35469c4698b4df829edbd9a14b0c28420d949eb7286047815a3102894c3" exitCode=60 Oct 31 15:45:20.237274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.211729 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerDied","Data":"81adc35469c4698b4df829edbd9a14b0c28420d949eb7286047815a3102894c3"} Oct 31 15:45:20.237274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.212165 4865 scope.go:122] "RemoveContainer" containerID="81adc35469c4698b4df829edbd9a14b0c28420d949eb7286047815a3102894c3" Oct 31 15:45:20.237274 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:20.212576 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h_odh-ai-gateway-infra(7c30a8f9-bb03-46d3-b427-af682644a990)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" Oct 31 15:45:20.237274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.236264 4865 scope.go:122] "RemoveContainer" containerID="1e653085c06e3f34b90643ef951e0a42b9f61269638c5d4ae5b0a0c487cd8691" Oct 31 15:45:20.257698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.257669 4865 scope.go:122] "RemoveContainer" containerID="0a1554f31a53d540d8d04b7d26e8fb0e3da8a00f480b9714f94cce3c37ca6efd" Oct 31 15:45:20.272826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.272798 4865 scope.go:122] "RemoveContainer" containerID="0739be0e7015b6fe85477c0e332bad31d092d14aae87112c6e843fbf20ce5b6e" Oct 31 15:45:20.375920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.375847 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:20.387298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.387268 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd9b917-d10e-44f8-b111-95c375a4ebc3-kube-api-access-krrkv\" (UniqueName: \"kubernetes.io/projected/9cd9b917-d10e-44f8-b111-95c375a4ebc3-kube-api-access-krrkv\") pod \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " Oct 31 15:45:20.387615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.387594 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd9b917-d10e-44f8-b111-95c375a4ebc3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9b917-d10e-44f8-b111-95c375a4ebc3-maas-api-service-ca\") pod \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\" (UID: \"9cd9b917-d10e-44f8-b111-95c375a4ebc3\") " Oct 31 15:45:20.387965 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.387940 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd9b917-d10e-44f8-b111-95c375a4ebc3-maas-api-service-ca" pod "9cd9b917-d10e-44f8-b111-95c375a4ebc3" (UID: "9cd9b917-d10e-44f8-b111-95c375a4ebc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:20.388236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.388126 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd9b917-d10e-44f8-b111-95c375a4ebc3-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.389731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.389704 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd9b917-d10e-44f8-b111-95c375a4ebc3-kube-api-access-krrkv" pod "9cd9b917-d10e-44f8-b111-95c375a4ebc3" (UID: "9cd9b917-d10e-44f8-b111-95c375a4ebc3"). InnerVolumeSpecName "kube-api-access-krrkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:20.411554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.411527 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:20.489709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.489674 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7ea082-ba12-4657-b084-bb96fa22de47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7ea082-ba12-4657-b084-bb96fa22de47-maas-api-service-ca\") pod \"be7ea082-ba12-4657-b084-bb96fa22de47\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " Oct 31 15:45:20.490211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.489886 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7ea082-ba12-4657-b084-bb96fa22de47-kube-api-access-x699z\" (UniqueName: \"kubernetes.io/projected/be7ea082-ba12-4657-b084-bb96fa22de47-kube-api-access-x699z\") pod \"be7ea082-ba12-4657-b084-bb96fa22de47\" (UID: \"be7ea082-ba12-4657-b084-bb96fa22de47\") " Oct 31 15:45:20.490211 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.490143 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7ea082-ba12-4657-b084-bb96fa22de47-maas-api-service-ca" pod "be7ea082-ba12-4657-b084-bb96fa22de47" (UID: "be7ea082-ba12-4657-b084-bb96fa22de47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:20.490433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.490409 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrkv\" (UniqueName: \"kubernetes.io/projected/9cd9b917-d10e-44f8-b111-95c375a4ebc3-kube-api-access-krrkv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.490552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.490438 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7ea082-ba12-4657-b084-bb96fa22de47-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.492565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.492531 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7ea082-ba12-4657-b084-bb96fa22de47-kube-api-access-x699z" pod "be7ea082-ba12-4657-b084-bb96fa22de47" (UID: "be7ea082-ba12-4657-b084-bb96fa22de47"). InnerVolumeSpecName "kube-api-access-x699z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:20.500458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.500430 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911d9a22-22da-4522-9836-303805eec788" path="/var/lib/kubelet/pods/911d9a22-22da-4522-9836-303805eec788/volumes" Oct 31 15:45:20.501018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.500965 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" path="/var/lib/kubelet/pods/c0a69a0e-1cbb-4cf4-982b-fae90350b824/volumes" Oct 31 15:45:20.591452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.591419 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x699z\" (UniqueName: \"kubernetes.io/projected/be7ea082-ba12-4657-b084-bb96fa22de47-kube-api-access-x699z\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.633380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.633308 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h"] Oct 31 15:45:20.760708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.760675 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4"] Oct 31 15:45:20.839756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.838066 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726"] Oct 31 15:45:20.843635 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.843603 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891025-hp726"] Oct 31 15:45:20.962641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.962618 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:20.998127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.998094 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0920ebf-b4dc-4bf1-80d4-577453362c54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0920ebf-b4dc-4bf1-80d4-577453362c54-maas-api-service-ca\") pod \"d0920ebf-b4dc-4bf1-80d4-577453362c54\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " Oct 31 15:45:20.998333 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.998316 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0920ebf-b4dc-4bf1-80d4-577453362c54-kube-api-access-j9dcm\" (UniqueName: \"kubernetes.io/projected/d0920ebf-b4dc-4bf1-80d4-577453362c54-kube-api-access-j9dcm\") pod \"d0920ebf-b4dc-4bf1-80d4-577453362c54\" (UID: \"d0920ebf-b4dc-4bf1-80d4-577453362c54\") " Oct 31 15:45:20.998547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.998523 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0920ebf-b4dc-4bf1-80d4-577453362c54-maas-api-service-ca" pod "d0920ebf-b4dc-4bf1-80d4-577453362c54" (UID: "d0920ebf-b4dc-4bf1-80d4-577453362c54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:20.998789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:20.998773 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0920ebf-b4dc-4bf1-80d4-577453362c54-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:21.000882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.000856 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0920ebf-b4dc-4bf1-80d4-577453362c54-kube-api-access-j9dcm" pod "d0920ebf-b4dc-4bf1-80d4-577453362c54" (UID: "d0920ebf-b4dc-4bf1-80d4-577453362c54"). InnerVolumeSpecName "kube-api-access-j9dcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:21.052257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.052220 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2"] Oct 31 15:45:21.055927 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.055897 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5a0cef-29891025-8pxz2"] Oct 31 15:45:21.100328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.100290 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9dcm\" (UniqueName: \"kubernetes.io/projected/d0920ebf-b4dc-4bf1-80d4-577453362c54-kube-api-access-j9dcm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:21.165423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.165393 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf"] Oct 31 15:45:21.168956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.168929 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891025-vbfxf"] Oct 31 15:45:21.209019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.208985 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 15:45:21.209175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.209030 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 15:45:21.221697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.221674 4865 generic.go:356] "Generic (PLEG): container finished" podID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerID="a0a3b0b05c01821bf97c892bdfb19c43f27ed577a9b00a24e9d63a002ae5e23b" exitCode=60 Oct 31 15:45:21.221867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.221749 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerDied","Data":"a0a3b0b05c01821bf97c892bdfb19c43f27ed577a9b00a24e9d63a002ae5e23b"} Oct 31 15:45:21.221867 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.221834 4865 scope.go:122] "RemoveContainer" containerID="6be5c9156ddf51dc4cadfe4dca7a10a228a7dcd5c07187a4c8cd6997cd508330" Oct 31 15:45:21.222190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.222172 4865 scope.go:122] "RemoveContainer" containerID="a0a3b0b05c01821bf97c892bdfb19c43f27ed577a9b00a24e9d63a002ae5e23b" Oct 31 15:45:21.222533 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:21.222507 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf_odh-ai-gateway-infra(d2247dcd-3c58-4437-81ef-b466a28710f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" Oct 31 15:45:21.225428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.225410 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" Oct 31 15:45:21.225428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.225417 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j" event={"ID":"be7ea082-ba12-4657-b084-bb96fa22de47","Type":"ContainerDied","Data":"4e5f86cfee539e96933f656b82e794e05ccfdeda44e22a138606d61735b3ef1b"} Oct 31 15:45:21.227083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.227061 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" event={"ID":"d0920ebf-b4dc-4bf1-80d4-577453362c54","Type":"ContainerDied","Data":"14fc5a5794a78553d60fe9825696dfa22c9a0efc880725899fa42fd467cef0d5"} Oct 31 15:45:21.227083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.227074 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4" Oct 31 15:45:21.231609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.231587 4865 generic.go:356] "Generic (PLEG): container finished" podID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerID="b1a3ce1c224f07779d677c472fe6abb56340627c36f7ef3ebc767063a45664f6" exitCode=60 Oct 31 15:45:21.231707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.231606 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" event={"ID":"ed090bb8-6b32-442a-b0f1-1169604b20cf","Type":"ContainerDied","Data":"b1a3ce1c224f07779d677c472fe6abb56340627c36f7ef3ebc767063a45664f6"} Oct 31 15:45:21.231975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.231957 4865 scope.go:122] "RemoveContainer" containerID="b1a3ce1c224f07779d677c472fe6abb56340627c36f7ef3ebc767063a45664f6" Oct 31 15:45:21.232213 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:21.232195 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s_odh-ai-gateway-infra(ed090bb8-6b32-442a-b0f1-1169604b20cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" Oct 31 15:45:21.241274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.241250 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" event={"ID":"9cd9b917-d10e-44f8-b111-95c375a4ebc3","Type":"ContainerDied","Data":"cea6d609317281de8c27432339d64c98475415c200c97658984d6edf9a527721"} Oct 31 15:45:21.241609 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.241435 4865 scope.go:122] "RemoveContainer" containerID="5cb8af0bbed38659867210692ca92220812ccc61bf83c77a54b153abb1ab451c" Oct 31 15:45:21.241793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.241554 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x" Oct 31 15:45:21.260770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.260646 4865 scope.go:122] "RemoveContainer" containerID="518a54655b3a43e9afefeaa73c5d8773b98d09013b70b478b2fa0ca0a00f77c7" Oct 31 15:45:21.275923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.275852 4865 scope.go:122] "RemoveContainer" containerID="1228ca51fb00cd15da07c487fa792c72ed348e9307efec043182c745f147b8bf" Oct 31 15:45:21.296659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.296583 4865 scope.go:122] "RemoveContainer" containerID="2f7e57bd05defdea291dba92e29c471e72c09cbb7a6988ae7db369b43140a414" Oct 31 15:45:21.409065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.409039 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:21.509516 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.509367 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b"] Oct 31 15:45:21.512817 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.512795 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891025-hv28b"] Oct 31 15:45:21.519636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.519615 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c30a8f9-bb03-46d3-b427-af682644a990-kube-api-access-csh9x\" (UniqueName: \"kubernetes.io/projected/7c30a8f9-bb03-46d3-b427-af682644a990-kube-api-access-csh9x\") pod \"7c30a8f9-bb03-46d3-b427-af682644a990\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " Oct 31 15:45:21.519751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.519659 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c30a8f9-bb03-46d3-b427-af682644a990-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30a8f9-bb03-46d3-b427-af682644a990-maas-api-service-ca\") pod \"7c30a8f9-bb03-46d3-b427-af682644a990\" (UID: \"7c30a8f9-bb03-46d3-b427-af682644a990\") " Oct 31 15:45:21.520161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.520137 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c30a8f9-bb03-46d3-b427-af682644a990-maas-api-service-ca" pod "7c30a8f9-bb03-46d3-b427-af682644a990" (UID: "7c30a8f9-bb03-46d3-b427-af682644a990"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:21.520528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.520502 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c30a8f9-bb03-46d3-b427-af682644a990-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:21.522162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.522139 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c30a8f9-bb03-46d3-b427-af682644a990-kube-api-access-csh9x" pod "7c30a8f9-bb03-46d3-b427-af682644a990" (UID: "7c30a8f9-bb03-46d3-b427-af682644a990"). InnerVolumeSpecName "kube-api-access-csh9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:21.561061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561027 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv"] Oct 31 15:45:21.561699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561684 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561700 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561711 4865 cpu_manager.go:414] "containerMap: removing container" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561716 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561732 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561737 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561745 4865 cpu_manager.go:414] "containerMap: removing container" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561749 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561755 4865 cpu_manager.go:414] "containerMap: removing container" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:21.561757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561760 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561775 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561780 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561788 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561793 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561800 4865 cpu_manager.go:414] "containerMap: removing container" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561804 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561811 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561816 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561825 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561829 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561836 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561840 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561847 4865 cpu_manager.go:414] "containerMap: removing container" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561851 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561858 4865 cpu_manager.go:414] "containerMap: removing container" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561862 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561874 4865 cpu_manager.go:414] "containerMap: removing container" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561878 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561887 4865 cpu_manager.go:414] "containerMap: removing container" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561891 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561897 4865 cpu_manager.go:414] "containerMap: removing container" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561901 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561907 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561911 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561919 4865 cpu_manager.go:414] "containerMap: removing container" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561923 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561931 4865 cpu_manager.go:414] "containerMap: removing container" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561935 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561940 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561944 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561950 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561954 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561966 4865 cpu_manager.go:414] "containerMap: removing container" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561970 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561977 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561981 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561991 4865 cpu_manager.go:414] "containerMap: removing container" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.561995 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562002 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562006 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562013 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:21.562036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562017 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562100 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562106 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562113 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562119 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562125 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562129 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562137 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562143 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562148 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562152 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562159 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562166 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562171 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562177 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562182 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562188 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562194 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562199 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562205 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562210 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562216 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562222 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562227 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:21.563677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.562232 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a69a0e-1cbb-4cf4-982b-fae90350b824" containerName="cleanup" Oct 31 15:45:21.565948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.565930 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.578048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.578019 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s"] Oct 31 15:45:21.621682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.621651 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.621682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.621704 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.621956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.621868 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:45:21.621956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.621935 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgtr\" (UniqueName: \"kubernetes.io/projected/62df08b0-3c14-4c6f-bed5-f416f361d618-kube-api-access-nlgtr\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.622165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.622080 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csh9x\" (UniqueName: \"kubernetes.io/projected/7c30a8f9-bb03-46d3-b427-af682644a990-kube-api-access-csh9x\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:21.623247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.623229 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv"] Oct 31 15:45:21.723002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.722971 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgtr\" (UniqueName: \"kubernetes.io/projected/62df08b0-3c14-4c6f-bed5-f416f361d618-kube-api-access-nlgtr\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.723159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.723060 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.723159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.723085 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.723930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.723903 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.725753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.725727 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.738805 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.738762 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j"] Oct 31 15:45:21.742660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.742638 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-24ca6e-29891025-22j5j"] Oct 31 15:45:21.780940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.780870 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgtr\" (UniqueName: \"kubernetes.io/projected/62df08b0-3c14-4c6f-bed5-f416f361d618-kube-api-access-nlgtr\") pod \"maas-api-revoke-sub-e0240de5-ll2xv\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.897419 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.897381 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:21.950226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.950195 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf"] Oct 31 15:45:21.978996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.978927 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x"] Oct 31 15:45:21.986025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:21.985983 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-06ec34-29891025-46h8x"] Oct 31 15:45:22.261689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.261658 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" event={"ID":"7c30a8f9-bb03-46d3-b427-af682644a990","Type":"ContainerDied","Data":"faa5bd40334b06c0a7b2d7766505a575256f8390cf8e5392b9115124f55d7dea"} Oct 31 15:45:22.261839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.261722 4865 scope.go:122] "RemoveContainer" containerID="81adc35469c4698b4df829edbd9a14b0c28420d949eb7286047815a3102894c3" Oct 31 15:45:22.261839 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.261814 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h" Oct 31 15:45:22.265991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.265960 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4"] Oct 31 15:45:22.272185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.272075 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4d1877-29891025-vkqg4"] Oct 31 15:45:22.284852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.284595 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv"] Oct 31 15:45:22.453557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.453522 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:22.508344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.508261 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" path="/var/lib/kubelet/pods/0058f3f3-372b-4879-bb51-d6a61e4fa4c7/volumes" Oct 31 15:45:22.508796 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.508777 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" path="/var/lib/kubelet/pods/210a8e6c-6ce3-48b1-9816-8c4842d06c64/volumes" Oct 31 15:45:22.509192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.509175 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d79372-75e0-419c-bff8-763faea71ff1" path="/var/lib/kubelet/pods/50d79372-75e0-419c-bff8-763faea71ff1/volumes" Oct 31 15:45:22.509869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.509851 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" path="/var/lib/kubelet/pods/53b8b14d-0d38-4e87-9d75-68f087d7ec1f/volumes" Oct 31 15:45:22.510791 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.510767 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd9b917-d10e-44f8-b111-95c375a4ebc3" path="/var/lib/kubelet/pods/9cd9b917-d10e-44f8-b111-95c375a4ebc3/volumes" Oct 31 15:45:22.511214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.511194 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" path="/var/lib/kubelet/pods/be7ea082-ba12-4657-b084-bb96fa22de47/volumes" Oct 31 15:45:22.512072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.512051 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" path="/var/lib/kubelet/pods/d0920ebf-b4dc-4bf1-80d4-577453362c54/volumes" Oct 31 15:45:22.534368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.534346 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:22.534760 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.534736 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2247dcd-3c58-4437-81ef-b466a28710f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2247dcd-3c58-4437-81ef-b466a28710f8-maas-api-service-ca\") pod \"d2247dcd-3c58-4437-81ef-b466a28710f8\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " Oct 31 15:45:22.534879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.534861 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2247dcd-3c58-4437-81ef-b466a28710f8-kube-api-access-j58sm\" (UniqueName: \"kubernetes.io/projected/d2247dcd-3c58-4437-81ef-b466a28710f8-kube-api-access-j58sm\") pod \"d2247dcd-3c58-4437-81ef-b466a28710f8\" (UID: \"d2247dcd-3c58-4437-81ef-b466a28710f8\") " Oct 31 15:45:22.535192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.535163 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2247dcd-3c58-4437-81ef-b466a28710f8-maas-api-service-ca" pod "d2247dcd-3c58-4437-81ef-b466a28710f8" (UID: "d2247dcd-3c58-4437-81ef-b466a28710f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:22.535854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.535833 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2247dcd-3c58-4437-81ef-b466a28710f8-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:22.538432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.538403 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2247dcd-3c58-4437-81ef-b466a28710f8-kube-api-access-j58sm" pod "d2247dcd-3c58-4437-81ef-b466a28710f8" (UID: "d2247dcd-3c58-4437-81ef-b466a28710f8"). InnerVolumeSpecName "kube-api-access-j58sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:22.636580 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.636531 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed090bb8-6b32-442a-b0f1-1169604b20cf-kube-api-access-lrv4h\" (UniqueName: \"kubernetes.io/projected/ed090bb8-6b32-442a-b0f1-1169604b20cf-kube-api-access-lrv4h\") pod \"ed090bb8-6b32-442a-b0f1-1169604b20cf\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " Oct 31 15:45:22.636771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.636707 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed090bb8-6b32-442a-b0f1-1169604b20cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed090bb8-6b32-442a-b0f1-1169604b20cf-maas-api-service-ca\") pod \"ed090bb8-6b32-442a-b0f1-1169604b20cf\" (UID: \"ed090bb8-6b32-442a-b0f1-1169604b20cf\") " Oct 31 15:45:22.637064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.637037 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed090bb8-6b32-442a-b0f1-1169604b20cf-maas-api-service-ca" pod "ed090bb8-6b32-442a-b0f1-1169604b20cf" (UID: "ed090bb8-6b32-442a-b0f1-1169604b20cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:22.637323 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.637305 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed090bb8-6b32-442a-b0f1-1169604b20cf-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:22.637444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.637328 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j58sm\" (UniqueName: \"kubernetes.io/projected/d2247dcd-3c58-4437-81ef-b466a28710f8-kube-api-access-j58sm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:22.639705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.639678 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed090bb8-6b32-442a-b0f1-1169604b20cf-kube-api-access-lrv4h" pod "ed090bb8-6b32-442a-b0f1-1169604b20cf" (UID: "ed090bb8-6b32-442a-b0f1-1169604b20cf"). InnerVolumeSpecName "kube-api-access-lrv4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:22.738764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.738726 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrv4h\" (UniqueName: \"kubernetes.io/projected/ed090bb8-6b32-442a-b0f1-1169604b20cf-kube-api-access-lrv4h\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:22.788113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.788035 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h"] Oct 31 15:45:22.791764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.791737 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891025-m949h"] Oct 31 15:45:22.926960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.926926 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:22.928206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.928179 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:22.929192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:22.929168 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 15:45:23.272219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.272188 4865 generic.go:356] "Generic (PLEG): container finished" podID="62df08b0-3c14-4c6f-bed5-f416f361d618" containerID="cd7bcc8cbfea4fe2a7d64bdd2b96794774ffacf3a8513a54de70abc17f8fc26a" exitCode=0 Oct 31 15:45:23.272353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.272272 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" event={"ID":"62df08b0-3c14-4c6f-bed5-f416f361d618","Type":"ContainerDied","Data":"cd7bcc8cbfea4fe2a7d64bdd2b96794774ffacf3a8513a54de70abc17f8fc26a"} Oct 31 15:45:23.272353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.272330 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" event={"ID":"62df08b0-3c14-4c6f-bed5-f416f361d618","Type":"ContainerStarted","Data":"a8711944dc7efb61b1ff8fbd6065c0f05dd74bf8d68d152a3d00e8f3d343d1b5"} Oct 31 15:45:23.273608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.273589 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" Oct 31 15:45:23.273703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.273619 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf" event={"ID":"d2247dcd-3c58-4437-81ef-b466a28710f8","Type":"ContainerDied","Data":"c1636b45ba048d7dcfbd33a9fcda69e7b29ecdbc8cac6f4c81b972e150c949dc"} Oct 31 15:45:23.273703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.273666 4865 scope.go:122] "RemoveContainer" containerID="a0a3b0b05c01821bf97c892bdfb19c43f27ed577a9b00a24e9d63a002ae5e23b" Oct 31 15:45:23.275110 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.275094 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" Oct 31 15:45:23.275182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.275122 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s" event={"ID":"ed090bb8-6b32-442a-b0f1-1169604b20cf","Type":"ContainerDied","Data":"ad11382745bc6e64d34e1862b1df588204d2b4a92856d64964e2f9b539e7dcc7"} Oct 31 15:45:23.284956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.284938 4865 scope.go:122] "RemoveContainer" containerID="b1a3ce1c224f07779d677c472fe6abb56340627c36f7ef3ebc767063a45664f6" Oct 31 15:45:23.501926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.501888 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s"] Oct 31 15:45:23.505050 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.505026 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-57815b-29891025-mqg8s"] Oct 31 15:45:23.616312 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.616230 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf"] Oct 31 15:45:23.619343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:23.619322 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-69517a-29891025-xncxf"] Oct 31 15:45:24.457166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.457144 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:24.497681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.497657 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" path="/var/lib/kubelet/pods/7c30a8f9-bb03-46d3-b427-af682644a990/volumes" Oct 31 15:45:24.498084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.498071 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" path="/var/lib/kubelet/pods/d2247dcd-3c58-4437-81ef-b466a28710f8/volumes" Oct 31 15:45:24.498462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.498449 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" path="/var/lib/kubelet/pods/ed090bb8-6b32-442a-b0f1-1169604b20cf/volumes" Oct 31 15:45:24.562628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.562552 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62df08b0-3c14-4c6f-bed5-f416f361d618-kube-api-access-nlgtr\" (UniqueName: \"kubernetes.io/projected/62df08b0-3c14-4c6f-bed5-f416f361d618-kube-api-access-nlgtr\") pod \"62df08b0-3c14-4c6f-bed5-f416f361d618\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " Oct 31 15:45:24.562628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.562597 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-openshift-ca\") pod \"62df08b0-3c14-4c6f-bed5-f416f361d618\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " Oct 31 15:45:24.562818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.562776 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-xks-ca\") pod \"62df08b0-3c14-4c6f-bed5-f416f361d618\" (UID: \"62df08b0-3c14-4c6f-bed5-f416f361d618\") " Oct 31 15:45:24.563038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.563009 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-openshift-ca" pod "62df08b0-3c14-4c6f-bed5-f416f361d618" (UID: "62df08b0-3c14-4c6f-bed5-f416f361d618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:24.563378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.563358 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:24.564931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.564907 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-xks-ca" pod "62df08b0-3c14-4c6f-bed5-f416f361d618" (UID: "62df08b0-3c14-4c6f-bed5-f416f361d618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:45:24.564931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.564914 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62df08b0-3c14-4c6f-bed5-f416f361d618-kube-api-access-nlgtr" pod "62df08b0-3c14-4c6f-bed5-f416f361d618" (UID: "62df08b0-3c14-4c6f-bed5-f416f361d618"). InnerVolumeSpecName "kube-api-access-nlgtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:24.664262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.664230 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlgtr\" (UniqueName: \"kubernetes.io/projected/62df08b0-3c14-4c6f-bed5-f416f361d618-kube-api-access-nlgtr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:24.664262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.664257 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62df08b0-3c14-4c6f-bed5-f416f361d618-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:24.892445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:24.892364 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:25.294800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:25.294766 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" event={"ID":"62df08b0-3c14-4c6f-bed5-f416f361d618","Type":"ContainerDied","Data":"a8711944dc7efb61b1ff8fbd6065c0f05dd74bf8d68d152a3d00e8f3d343d1b5"} Oct 31 15:45:25.294967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:25.294823 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8711944dc7efb61b1ff8fbd6065c0f05dd74bf8d68d152a3d00e8f3d343d1b5" Oct 31 15:45:25.294967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:25.294774 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv" Oct 31 15:45:26.994651 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.994616 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2"] Oct 31 15:45:26.995286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995271 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995287 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995299 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995303 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995310 4865 cpu_manager.go:414] "containerMap: removing container" podUID="62df08b0-3c14-4c6f-bed5-f416f361d618" containerName="revoke-keys" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995314 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62df08b0-3c14-4c6f-bed5-f416f361d618" containerName="revoke-keys" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995321 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995325 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c30a8f9-bb03-46d3-b427-af682644a990" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995339 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995343 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:45:26.995346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995349 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995354 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995369 4865 cpu_manager.go:414] "containerMap: removing container" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995373 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b8b14d-0d38-4e87-9d75-68f087d7ec1f" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995388 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995392 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995401 4865 cpu_manager.go:414] "containerMap: removing container" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995405 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995527 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995538 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995555 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995563 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="911d9a22-22da-4522-9836-303805eec788" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995574 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995583 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7ea082-ba12-4657-b084-bb96fa22de47" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995590 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0920ebf-b4dc-4bf1-80d4-577453362c54" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995595 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d79372-75e0-419c-bff8-763faea71ff1" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995602 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="210a8e6c-6ce3-48b1-9816-8c4842d06c64" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995607 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995613 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995619 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="62df08b0-3c14-4c6f-bed5-f416f361d618" containerName="revoke-keys" Oct 31 15:45:26.995692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:26.995625 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0058f3f3-372b-4879-bb51-d6a61e4fa4c7" containerName="cleanup" Oct 31 15:45:27.000606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.000585 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.098638 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.098607 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.098781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.098645 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.098848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.098832 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fa31b0-d628-4102-9afb-2755cf2339cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.098884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.098874 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.098992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.098974 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.099027 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.099014 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.099095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.099070 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnsk\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-kube-api-access-hvnsk\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.099202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.099175 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.099342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.099213 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.198208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.198173 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2"] Oct 31 15:45:27.200457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200433 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200512 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnsk\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-kube-api-access-hvnsk\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200537 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200554 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200608 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200642 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200819 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fa31b0-d628-4102-9afb-2755cf2339cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200874 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.200977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.200945 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.201268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.201243 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.201379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.201243 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.201499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.201459 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.201727 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.201702 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.202154 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.202126 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fa31b0-d628-4102-9afb-2755cf2339cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.204189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.204166 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.204455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.204433 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.261840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.261775 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.261840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.261824 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnsk\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-kube-api-access-hvnsk\") pod \"maas-default-gateway-openshift-default-67bf986f99-q54j2\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.314513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.314463 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:27.744756 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:45:27.744721 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fa31b0_d628_4102_9afb_2755cf2339cc.slice/crio-383d1604a9b47cd7da4c32387f50cb74a0b1bc7a2a7971635db36db296284804 WatchSource:0}: Error finding container 383d1604a9b47cd7da4c32387f50cb74a0b1bc7a2a7971635db36db296284804: Status 404 returned error can't find the container with id 383d1604a9b47cd7da4c32387f50cb74a0b1bc7a2a7971635db36db296284804 Oct 31 15:45:27.746547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.746524 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2"] Oct 31 15:45:27.748347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.748309 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:27.748430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.748381 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:27.748430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:27.748420 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:45:28.319746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:28.319707 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" event={"ID":"76fa31b0-d628-4102-9afb-2755cf2339cc","Type":"ContainerStarted","Data":"99e5055bf12a74a16bb3f0ca718fbd8767f6c35720a250bd1e8e1ae810a6a682"} Oct 31 15:45:28.320164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:28.319776 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" event={"ID":"76fa31b0-d628-4102-9afb-2755cf2339cc","Type":"ContainerStarted","Data":"383d1604a9b47cd7da4c32387f50cb74a0b1bc7a2a7971635db36db296284804"} Oct 31 15:45:28.440565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:28.440504 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podStartSLOduration=2.440489161 podStartE2EDuration="2.440489161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:28.436922897 +0000 UTC m=+3574.056474310" watchObservedRunningTime="2026-10-31 15:45:28.440489161 +0000 UTC m=+3574.060040572" Oct 31 15:45:29.319914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:29.319875 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:29.324693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:29.324660 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 31 15:45:29.324869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:29.324717 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 31 15:45:29.636044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:29.635910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:30.318497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:30.316665 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 31 15:45:30.318497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:30.316729 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 31 15:45:31.319625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:31.319586 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 31 15:45:31.320152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:31.320026 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 31 15:45:32.315800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:32.315759 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 31 15:45:32.316065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:32.316037 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 31 15:45:33.320179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:33.320143 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 31 15:45:33.320837 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:33.320796 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 31 15:45:34.315716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:34.315678 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 31 15:45:34.315914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:34.315740 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 31 15:45:34.411373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:34.411333 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:35.246584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:35.246546 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:35.315152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:35.315115 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body= Oct 31 15:45:35.315374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:35.315189 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" Oct 31 15:45:37.270286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:37.270229 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:37.270933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:37.270677 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:37.271773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:37.271749 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 15:45:39.113009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:39.112973 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:39.829825 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:39.829790 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:45:42.768345 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:42.768302 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:42.887795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:42.887762 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:43.163443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:43.163355 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:44.865083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:44.865044 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:47.186055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:47.186026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:47.321764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:47.321726 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:47.491229 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:45:47.491195 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:45:48.702884 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:48.702848 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:49.178685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:49.178654 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:49.646941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:49.646910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:51.357229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:51.357198 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:51.928270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:51.928240 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:52.205321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:52.205240 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:54.577187 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:54.577119 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:55.909141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:55.909106 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:56.073960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:56.073923 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:56.077880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:56.077858 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:45:56.086023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:56.086004 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:45:56.221164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:56.221133 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:58.520976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:58.520949 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:45:58.529941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:58.529921 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:45:59.106101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:59.106065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:59.301903 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:59.301872 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:45:59.425840 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:45:59.425812 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:01.008360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:01.008324 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:02.049415 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:02.049373 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:02.171866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:02.171829 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:03.594603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:03.594562 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:04.092267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:04.092235 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj"] Oct 31 15:46:04.092519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:04.092496 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" containerID="cri-o://845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf" gracePeriod=30 Oct 31 15:46:04.115111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:04.115081 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:04.855549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:04.855518 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:05.520747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:05.520715 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:06.030647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:06.030618 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:06.295592 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:06.295510 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:06.906494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:06.901404 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:07.029893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:07.029855 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:07.294259 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:07.294217 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:08.321044 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:08.320997 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:08.452449 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:08.452417 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:08.633384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:08.633303 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:09.374315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.374247 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:46:09.464985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.464953 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-podinfo\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465000 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-envoy\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465020 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-credential-socket\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465134 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-socket\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465219 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e3f80f-a50e-4b21-87da-4e069cf41144-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e3f80f-a50e-4b21-87da-4e069cf41144-istiod-ca-cert\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465294 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-certs\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465501 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-kube-api-access-r97kc\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-kube-api-access-r97kc\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465591 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-token\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-token\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.465739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.465706 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-data\") pod \"c9e3f80f-a50e-4b21-87da-4e069cf41144\" (UID: \"c9e3f80f-a50e-4b21-87da-4e069cf41144\") " Oct 31 15:46:09.466219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466035 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-credential-socket" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:46:09.466219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466165 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-certs" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:46:09.466219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466166 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e3f80f-a50e-4b21-87da-4e069cf41144-istiod-ca-cert" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:46:09.466566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466332 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-data" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:46:09.466566 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466392 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-socket" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:46:09.466758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466737 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.466810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466766 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.466810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466782 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.466810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466797 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e3f80f-a50e-4b21-87da-4e069cf41144-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.466918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.466811 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.467996 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.467970 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-podinfo" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:46:09.468150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.468132 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-envoy" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:46:09.468383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.468366 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-token" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:46:09.468532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.468512 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-kube-api-access-r97kc" pod "c9e3f80f-a50e-4b21-87da-4e069cf41144" (UID: "c9e3f80f-a50e-4b21-87da-4e069cf41144"). InnerVolumeSpecName "kube-api-access-r97kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:46:09.567858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.567767 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r97kc\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-kube-api-access-r97kc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.567858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.567800 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.567858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.567814 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.567858 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.567826 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e3f80f-a50e-4b21-87da-4e069cf41144-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:09.625384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.625302 4865 generic.go:356] "Generic (PLEG): container finished" podID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerID="845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf" exitCode=0 Oct 31 15:46:09.625384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.625350 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" Oct 31 15:46:09.625661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.625390 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" event={"ID":"c9e3f80f-a50e-4b21-87da-4e069cf41144","Type":"ContainerDied","Data":"845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf"} Oct 31 15:46:09.625661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.625443 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj" event={"ID":"c9e3f80f-a50e-4b21-87da-4e069cf41144","Type":"ContainerDied","Data":"62aad49a838a464375f3ae7cc71580250657a3445ed0a9e936fe05fa44cf2ead"} Oct 31 15:46:09.625661 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.625499 4865 scope.go:122] "RemoveContainer" containerID="845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf" Oct 31 15:46:09.636882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.636862 4865 scope.go:122] "RemoveContainer" containerID="9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843" Oct 31 15:46:09.646748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.646733 4865 scope.go:122] "RemoveContainer" containerID="845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf" Oct 31 15:46:09.647020 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:46:09.647001 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf\": container with ID starting with 845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf not found: ID does not exist" containerID="845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf" Oct 31 15:46:09.647086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.647029 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf"} err="failed to get container status \"845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf\": rpc error: code = NotFound desc = could not find container \"845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf\": container with ID starting with 845e005d64760a35b911a1439f7f5545a964ebe8c5566ce7e00de08bfc46a0bf not found: ID does not exist" Oct 31 15:46:09.647086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.647052 4865 scope.go:122] "RemoveContainer" containerID="9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843" Oct 31 15:46:09.647260 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:46:09.647240 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843\": container with ID starting with 9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843 not found: ID does not exist" containerID="9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843" Oct 31 15:46:09.647320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.647271 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843"} err="failed to get container status \"9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843\": rpc error: code = NotFound desc = could not find container \"9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843\": container with ID starting with 9d5fad41fa2b2c27f1ecc651ce6ba11579c0b8c8276d56f447212d64fc044843 not found: ID does not exist" Oct 31 15:46:09.793225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.793193 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj"] Oct 31 15:46:09.801458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:09.801433 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-4d1877-openshift-default-77d54bfcf7-btvgj"] Oct 31 15:46:10.497218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:10.497183 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" path="/var/lib/kubelet/pods/c9e3f80f-a50e-4b21-87da-4e069cf41144/volumes" Oct 31 15:46:10.709788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:10.709756 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:11.187625 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:11.187591 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:11.632738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:11.632696 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v"] Oct 31 15:46:11.869398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:11.869366 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:11.988115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:11.988084 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:15.738813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:15.738783 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:16.633455 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:16.633418 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" podUID="d7b67d0c-a8c4-42bd-9b15-757809b4447b" containerName="payload-pre-processing" containerID="cri-o://97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d" gracePeriod=25 Oct 31 15:46:16.702295 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:16.702265 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:16.920685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:16.920654 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:46:17.065958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.065924 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b67d0c-a8c4-42bd-9b15-757809b4447b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b67d0c-a8c4-42bd-9b15-757809b4447b-plugins-config-volume\") pod \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " Oct 31 15:46:17.066153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.066009 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b67d0c-a8c4-42bd-9b15-757809b4447b-kube-api-access-c5lfq\" (UniqueName: \"kubernetes.io/projected/d7b67d0c-a8c4-42bd-9b15-757809b4447b-kube-api-access-c5lfq\") pod \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\" (UID: \"d7b67d0c-a8c4-42bd-9b15-757809b4447b\") " Oct 31 15:46:17.066340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.066309 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b67d0c-a8c4-42bd-9b15-757809b4447b-plugins-config-volume" pod "d7b67d0c-a8c4-42bd-9b15-757809b4447b" (UID: "d7b67d0c-a8c4-42bd-9b15-757809b4447b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:46:17.066692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.066670 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b67d0c-a8c4-42bd-9b15-757809b4447b-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:17.068510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.068480 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b67d0c-a8c4-42bd-9b15-757809b4447b-kube-api-access-c5lfq" pod "d7b67d0c-a8c4-42bd-9b15-757809b4447b" (UID: "d7b67d0c-a8c4-42bd-9b15-757809b4447b"). InnerVolumeSpecName "kube-api-access-c5lfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:46:17.168207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.168111 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lfq\" (UniqueName: \"kubernetes.io/projected/d7b67d0c-a8c4-42bd-9b15-757809b4447b-kube-api-access-c5lfq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:17.490051 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:46:17.490016 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:46:17.687060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.687026 4865 generic.go:356] "Generic (PLEG): container finished" podID="d7b67d0c-a8c4-42bd-9b15-757809b4447b" containerID="97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d" exitCode=0 Oct 31 15:46:17.687263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.687070 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" Oct 31 15:46:17.687263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.687108 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" event={"ID":"d7b67d0c-a8c4-42bd-9b15-757809b4447b","Type":"ContainerDied","Data":"97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d"} Oct 31 15:46:17.687263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.687164 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v" event={"ID":"d7b67d0c-a8c4-42bd-9b15-757809b4447b","Type":"ContainerDied","Data":"03770401a3b8e9d05f44659e69f4cb37876cbec3d5dcfc4b813c704d88905b24"} Oct 31 15:46:17.687263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.687191 4865 scope.go:122] "RemoveContainer" containerID="97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d" Oct 31 15:46:17.700396 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.700377 4865 scope.go:122] "RemoveContainer" containerID="97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d" Oct 31 15:46:17.700698 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:46:17.700676 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d\": container with ID starting with 97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d not found: ID does not exist" containerID="97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d" Oct 31 15:46:17.700800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.700704 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d"} err="failed to get container status \"97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d\": rpc error: code = NotFound desc = could not find container \"97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d\": container with ID starting with 97825c78e542e8cbffaf8d1afde259e0ee167cc5ba77dd214309c9f829e22f7d not found: ID does not exist" Oct 31 15:46:17.854099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.854014 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v"] Oct 31 15:46:17.905101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.905072 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69517a-65fb55d567-rrr8v"] Oct 31 15:46:17.995941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:17.995904 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:18.497970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:18.497938 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b67d0c-a8c4-42bd-9b15-757809b4447b" path="/var/lib/kubelet/pods/d7b67d0c-a8c4-42bd-9b15-757809b4447b/volumes" Oct 31 15:46:19.370051 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:19.370021 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:20.895958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:20.895921 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:21.318698 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:21.318661 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:24.961952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:24.961918 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:26.682459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:26.682427 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:30.888035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:30.888008 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:31.331826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:31.331759 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:31.457976 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:31.457942 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:31.637225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:31.637132 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:31.769586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:31.769546 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:32.208895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.208861 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:32.722505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.722457 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn"] Oct 31 15:46:32.723245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723222 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" Oct 31 15:46:32.723245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723244 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" Oct 31 15:46:32.723459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723281 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d7b67d0c-a8c4-42bd-9b15-757809b4447b" containerName="payload-pre-processing" Oct 31 15:46:32.723459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723288 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b67d0c-a8c4-42bd-9b15-757809b4447b" containerName="payload-pre-processing" Oct 31 15:46:32.723459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723332 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:46:32.723459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723339 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2247dcd-3c58-4437-81ef-b466a28710f8" containerName="cleanup" Oct 31 15:46:32.723459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723359 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:46:32.723459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723366 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed090bb8-6b32-442a-b0f1-1169604b20cf" containerName="cleanup" Oct 31 15:46:32.723843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723558 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" Oct 31 15:46:32.723843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723579 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b67d0c-a8c4-42bd-9b15-757809b4447b" containerName="payload-pre-processing" Oct 31 15:46:32.723843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.723592 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" Oct 31 15:46:32.727306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.727286 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784357 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22249730-1bc1-459b-b69a-5c83060c98c2-istiod-ca-cert\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784429 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-credential-socket\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784483 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-envoy\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784539 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bgg\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-kube-api-access-r8bgg\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784571 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-socket\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784669 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-istio-token\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784706 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22249730-1bc1-459b-b69a-5c83060c98c2-istio-podinfo\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784742 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-data\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.784835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.784781 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-certs\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.789756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.789728 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-73b8d8-openshift-default-dockercfg-z86sk\"" Oct 31 15:46:32.862153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.861998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:32.874591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.874446 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn"] Oct 31 15:46:32.888294 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.888236 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22249730-1bc1-459b-b69a-5c83060c98c2-istiod-ca-cert\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.888584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.888558 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-credential-socket\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.888780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.888758 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-envoy\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.888900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.888829 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bgg\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-kube-api-access-r8bgg\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.888900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.888866 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-socket\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889009 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.888976 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-istio-token\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889019 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22249730-1bc1-459b-b69a-5c83060c98c2-istio-podinfo\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889030 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-credential-socket\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889090 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889054 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-data\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889114 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-certs\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889307 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-socket\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889513 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-certs\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889521 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22249730-1bc1-459b-b69a-5c83060c98c2-istiod-ca-cert\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.889643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.889557 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-data\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.892570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.892531 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22249730-1bc1-459b-b69a-5c83060c98c2-istio-podinfo\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.893216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.893193 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-envoy\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.948645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.948605 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-istio-token\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:32.948860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:32.948830 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bgg\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-kube-api-access-r8bgg\") pod \"e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:33.041714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.041634 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:33.181318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.181272 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:33.470536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.470506 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn"] Oct 31 15:46:33.471595 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:46:33.471565 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22249730_1bc1_459b_b69a_5c83060c98c2.slice/crio-675ceaa83b23d510375a8406a31778f0f6a40984ff9ba9b8681feb07e6894aee WatchSource:0}: Error finding container 675ceaa83b23d510375a8406a31778f0f6a40984ff9ba9b8681feb07e6894aee: Status 404 returned error can't find the container with id 675ceaa83b23d510375a8406a31778f0f6a40984ff9ba9b8681feb07e6894aee Oct 31 15:46:33.474245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.474213 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:46:33.474336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.474290 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:46:33.474336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.474319 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:46:33.865970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.865876 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" event={"ID":"22249730-1bc1-459b-b69a-5c83060c98c2","Type":"ContainerStarted","Data":"0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5"} Oct 31 15:46:33.865970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.865940 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" event={"ID":"22249730-1bc1-459b-b69a-5c83060c98c2","Type":"ContainerStarted","Data":"675ceaa83b23d510375a8406a31778f0f6a40984ff9ba9b8681feb07e6894aee"} Oct 31 15:46:33.986298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:33.986221 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podStartSLOduration=1.9862051250000001 podStartE2EDuration="1.986205125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:33.984040089 +0000 UTC m=+3639.603591501" watchObservedRunningTime="2026-10-31 15:46:33.986205125 +0000 UTC m=+3639.605756583" Oct 31 15:46:34.042132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:34.042097 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:34.048145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:34.048117 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:34.521515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:34.515276 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:34.642667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:34.642618 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:34.872186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:34.872102 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:34.873034 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:34.873015 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:46:36.152117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:36.152036 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:36.863878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:36.863833 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:37.127586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:37.127501 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:37.303782 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:37.303752 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:37.437759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:37.437667 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:39.741751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:39.741717 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:46:40.337320 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:40.337288 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:40.503331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:40.503296 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:41.987535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:41.987464 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:46.916033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:46.916003 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:47.491168 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:46:47.491128 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:46:51.605014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:51.604981 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:54.679605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:54.679564 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:46:56.304808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:46:56.304772 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:00.981406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:00.981365 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:05.677962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:05.677924 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:05.860690 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:05.860648 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:06.848833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:06.848804 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:47:07.396408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:07.396374 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:10.319434 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:10.319397 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:14.919326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:14.919288 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:15.088334 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:15.088297 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:15.219222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:15.219186 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:16.499164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:16.499128 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:16.818919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:16.818831 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:16.952959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:16.952927 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:17.491657 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:47:17.491620 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:47:18.261382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:18.261346 4865 generic.go:356] "Generic (PLEG): container finished" podID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerID="eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723" exitCode=0 Oct 31 15:47:18.261862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:18.261426 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" event={"ID":"a507b21c-dc8b-4741-a732-a5a3cb96f161","Type":"ContainerDied","Data":"eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723"} Oct 31 15:47:18.261925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:18.261864 4865 scope.go:122] "RemoveContainer" containerID="eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723" Oct 31 15:47:18.263436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:18.263405 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:47:18.263531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:18.263496 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:47:18.263577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:18.263531 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:47:19.270498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:19.270436 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" event={"ID":"a507b21c-dc8b-4741-a732-a5a3cb96f161","Type":"ContainerStarted","Data":"41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987"} Oct 31 15:47:19.515492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:19.515164 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:47:19.518615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:19.518570 4865 patch_prober.go:29] interesting pod/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 15:47:19.518794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:19.518632 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 15:47:19.814437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:19.814400 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:20.519664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:20.519623 4865 patch_prober.go:29] interesting pod/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 15:47:20.520316 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:20.520280 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 15:47:21.516913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:21.516873 4865 patch_prober.go:29] interesting pod/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 15:47:21.517219 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:21.517185 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 15:47:21.981910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:21.981874 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:22.516427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:22.516389 4865 patch_prober.go:29] interesting pod/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 15:47:22.516934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:22.516809 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 15:47:23.028629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:23.028583 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:23.523889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:23.523851 4865 patch_prober.go:29] interesting pod/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 15:47:23.524177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:23.524146 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 15:47:24.554442 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:24.554404 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:47:24.555431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:24.555377 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:47:24.557418 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:24.557397 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:47:24.606382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:24.606347 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:29.237721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:29.237681 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:33.325290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:33.325243 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:35.032362 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:35.032305 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:36.359756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:36.359720 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:36.853811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:36.853739 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:37.740589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:37.740553 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:38.643517 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:38.642026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:38.871010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:38.870905 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:40.175864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:40.175828 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:40.597933 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:40.597899 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:40.720642 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:40.720595 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:42.381283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:42.381248 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:42.552399 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:42.552366 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:42.671764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:42.671685 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:45.234636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:45.234598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:46.359613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:46.359568 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:46.538205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:46.538165 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:47.492051 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:47:47.492014 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:47:49.887971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:49.887911 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:54.416311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:54.416280 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:54.571826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:54.571789 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:54.686771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:54.686677 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:57.365943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:57.365909 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:47:58.859498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:58.858867 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:59.365528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:59.365499 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:59.649792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:59.649706 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:47:59.917492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:47:59.917394 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:00.109194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:00.109151 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:00.242006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:00.241954 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:00.519951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:00.519865 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:01.623942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.623910 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc"] Oct 31 15:48:01.624699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.624680 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" Oct 31 15:48:01.624755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.624701 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e3f80f-a50e-4b21-87da-4e069cf41144" containerName="istio-proxy" Oct 31 15:48:01.628358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.628339 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.677429 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677391 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-certs\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.677658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677456 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-credential-socket\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.677658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677606 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830ca810-fbe2-4e37-a51b-35b66c534776-istio-podinfo\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.677795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677660 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-data\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.677795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677695 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-istio-token\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.677795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677751 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830ca810-fbe2-4e37-a51b-35b66c534776-istiod-ca-cert\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.677795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677783 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-socket\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.678039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.677973 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwq2\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-kube-api-access-5pwq2\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.678039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.678010 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-envoy\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.709134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.709104 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:01.728102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.728069 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc"] Oct 31 15:48:01.737682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.737656 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4c9a78-openshift-default-dockercfg-gjwhg\"" Oct 31 15:48:01.786653 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.786615 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwq2\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-kube-api-access-5pwq2\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.789809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.789765 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-envoy\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.798045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798016 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-certs\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798252 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-credential-socket\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798380 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830ca810-fbe2-4e37-a51b-35b66c534776-istio-podinfo\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798434 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-data\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798482 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-istio-token\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798531 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830ca810-fbe2-4e37-a51b-35b66c534776-istiod-ca-cert\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798565 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-socket\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.798622 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-certs\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.799492 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830ca810-fbe2-4e37-a51b-35b66c534776-istiod-ca-cert\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.799740 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-socket\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.800394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.799968 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-data\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.802161 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.802095 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-credential-socket\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.813098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.813037 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-envoy\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.823078 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.823047 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830ca810-fbe2-4e37-a51b-35b66c534776-istio-podinfo\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.844572 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.844547 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwq2\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-kube-api-access-5pwq2\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.858697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.858667 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-istio-token\") pod \"e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:01.943937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:01.943869 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:02.014857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.010693 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:02.385562 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:48:02.385531 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830ca810_fbe2_4e37_a51b_35b66c534776.slice/crio-4f75aabce7a664f78ed48ba6759a0187e74a81437161a9e12a7c63504b5bcb29 WatchSource:0}: Error finding container 4f75aabce7a664f78ed48ba6759a0187e74a81437161a9e12a7c63504b5bcb29: Status 404 returned error can't find the container with id 4f75aabce7a664f78ed48ba6759a0187e74a81437161a9e12a7c63504b5bcb29 Oct 31 15:48:02.385795 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.385776 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc"] Oct 31 15:48:02.388236 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.388200 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:48:02.388350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.388269 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:48:02.388350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.388300 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:48:02.634105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.634008 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" event={"ID":"830ca810-fbe2-4e37-a51b-35b66c534776","Type":"ContainerStarted","Data":"d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3"} Oct 31 15:48:02.634105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.634086 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" event={"ID":"830ca810-fbe2-4e37-a51b-35b66c534776","Type":"ContainerStarted","Data":"4f75aabce7a664f78ed48ba6759a0187e74a81437161a9e12a7c63504b5bcb29"} Oct 31 15:48:02.752278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.752216 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podStartSLOduration=1.752201691 podStartE2EDuration="1.752201691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:02.751161581 +0000 UTC m=+3728.370713005" watchObservedRunningTime="2026-10-31 15:48:02.752201691 +0000 UTC m=+3728.371753102" Oct 31 15:48:02.957519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:02.956626 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:03.547140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:03.547104 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:03.719519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:03.719455 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:03.857282 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:03.857196 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:03.950425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:03.950372 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:04.648268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:04.648233 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:04.649130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:04.649110 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:48:04.789298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:04.789257 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:06.186504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:06.186420 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:06.322094 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:06.322061 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:06.473072 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:06.473039 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:07.748411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:07.748376 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:09.180234 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.180199 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:09.356504 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.356070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:09.599704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.599670 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj"] Oct 31 15:48:09.604241 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.604217 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:09.660292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.660269 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj"] Oct 31 15:48:09.706120 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.706088 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c4840c\"" Oct 31 15:48:09.706675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.706603 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c4840c-dockercfg-hrtk7\"" Oct 31 15:48:09.750600 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.741341 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:09.781700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.781665 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95m2t\" (UniqueName: \"kubernetes.io/projected/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-kube-api-access-95m2t\") pod \"payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:09.884897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.884810 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:09.885089 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.884910 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95m2t\" (UniqueName: \"kubernetes.io/projected/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-kube-api-access-95m2t\") pod \"payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:09.885619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.885594 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:09.942896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:09.942867 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95m2t\" (UniqueName: \"kubernetes.io/projected/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-kube-api-access-95m2t\") pod \"payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:10.012527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:10.012494 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:10.219708 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:10.219672 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:10.595068 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:48:10.595036 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748e3dee_d8d1_4fe0_ae01_4d1f8b87fd5f.slice/crio-2434379b9a25aeb6fdb6bf7e02cda3b4ada0735b0f6b5ca7151c676e8594cb67 WatchSource:0}: Error finding container 2434379b9a25aeb6fdb6bf7e02cda3b4ada0735b0f6b5ca7151c676e8594cb67: Status 404 returned error can't find the container with id 2434379b9a25aeb6fdb6bf7e02cda3b4ada0735b0f6b5ca7151c676e8594cb67 Oct 31 15:48:10.595701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:10.595628 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj"] Oct 31 15:48:10.753349 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:10.753310 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" event={"ID":"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f","Type":"ContainerStarted","Data":"cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6"} Oct 31 15:48:10.753584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:10.753371 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" event={"ID":"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f","Type":"ContainerStarted","Data":"2434379b9a25aeb6fdb6bf7e02cda3b4ada0735b0f6b5ca7151c676e8594cb67"} Oct 31 15:48:10.753584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:10.753397 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:10.867773 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:10.867662 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" podStartSLOduration=1.8676455280000002 podStartE2EDuration="1.867645528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:10.867525006 +0000 UTC m=+3736.487076419" watchObservedRunningTime="2026-10-31 15:48:10.867645528 +0000 UTC m=+3736.487196942" Oct 31 15:48:11.703569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:11.703539 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:11.882403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:11.879883 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:12.121358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.121327 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq"] Oct 31 15:48:12.125779 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.125761 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.180990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.180966 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq"] Oct 31 15:48:12.239126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.228980 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4c9a78-dockercfg-5rjk2\"" Oct 31 15:48:12.239126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.229593 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4c9a78\"" Oct 31 15:48:12.239126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.231185 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht66c\" (UniqueName: \"kubernetes.io/projected/504e7fdf-388b-4db6-951f-376b2117bd22-kube-api-access-ht66c\") pod \"payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.239126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.231260 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e7fdf-388b-4db6-951f-376b2117bd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.341491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.338372 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht66c\" (UniqueName: \"kubernetes.io/projected/504e7fdf-388b-4db6-951f-376b2117bd22-kube-api-access-ht66c\") pod \"payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.341491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.338462 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e7fdf-388b-4db6-951f-376b2117bd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.341491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.339396 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e7fdf-388b-4db6-951f-376b2117bd22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.396689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.396615 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht66c\" (UniqueName: \"kubernetes.io/projected/504e7fdf-388b-4db6-951f-376b2117bd22-kube-api-access-ht66c\") pod \"payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.439877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.439847 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:12.818431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:12.818404 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq"] Oct 31 15:48:12.819390 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:48:12.819363 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504e7fdf_388b_4db6_951f_376b2117bd22.slice/crio-ca3a667487282c1cb3b33121b8e7efec4a5d52c5bcda32fbe84abec13e7c3a60 WatchSource:0}: Error finding container ca3a667487282c1cb3b33121b8e7efec4a5d52c5bcda32fbe84abec13e7c3a60: Status 404 returned error can't find the container with id ca3a667487282c1cb3b33121b8e7efec4a5d52c5bcda32fbe84abec13e7c3a60 Oct 31 15:48:13.778380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:13.778348 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" event={"ID":"504e7fdf-388b-4db6-951f-376b2117bd22","Type":"ContainerStarted","Data":"9dd2710b5fa910813d47af42beb1fda24f6b016cc12b5a05092d4129558453b1"} Oct 31 15:48:13.778612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:13.778399 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" event={"ID":"504e7fdf-388b-4db6-951f-376b2117bd22","Type":"ContainerStarted","Data":"ca3a667487282c1cb3b33121b8e7efec4a5d52c5bcda32fbe84abec13e7c3a60"} Oct 31 15:48:13.778612 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:13.778439 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:13.902226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:13.902171 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" podStartSLOduration=1.902157673 podStartE2EDuration="1.902157673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:13.900402476 +0000 UTC m=+3739.519953888" watchObservedRunningTime="2026-10-31 15:48:13.902157673 +0000 UTC m=+3739.521709098" Oct 31 15:48:15.012701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:15.012671 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:17.492100 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:48:17.492064 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:48:19.685519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:19.685458 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:21.763819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:21.763791 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:48:24.405697 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:24.405660 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:24.788136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:24.788107 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:48:26.897619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:26.897584 4865 generic.go:356] "Generic (PLEG): container finished" podID="22249730-1bc1-459b-b69a-5c83060c98c2" containerID="0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5" exitCode=0 Oct 31 15:48:26.898067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:26.897659 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" event={"ID":"22249730-1bc1-459b-b69a-5c83060c98c2","Type":"ContainerDied","Data":"0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5"} Oct 31 15:48:26.898112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:26.898103 4865 scope.go:122] "RemoveContainer" containerID="0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5" Oct 31 15:48:26.899768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:26.899742 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:48:26.899834 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:26.899810 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:48:26.899872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:26.899845 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:48:27.908175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:27.908137 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" event={"ID":"22249730-1bc1-459b-b69a-5c83060c98c2","Type":"ContainerStarted","Data":"96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f"} Oct 31 15:48:28.042615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:28.042576 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:48:28.042913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:28.042888 4865 patch_prober.go:29] interesting pod/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 31 15:48:28.042961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:28.042945 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 31 15:48:29.042427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:29.042388 4865 patch_prober.go:29] interesting pod/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 31 15:48:29.042878 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:29.042445 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 31 15:48:29.119308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:29.119272 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:30.043007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:30.042971 4865 patch_prober.go:29] interesting pod/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 31 15:48:30.043431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:30.043031 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 31 15:48:31.042556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:31.042517 4865 patch_prober.go:29] interesting pod/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 31 15:48:31.042556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:31.042576 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 31 15:48:32.042652 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:32.042616 4865 patch_prober.go:29] interesting pod/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 31 15:48:32.043111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:32.042674 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 31 15:48:33.042934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:33.042891 4865 patch_prober.go:29] interesting pod/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body= Oct 31 15:48:33.043431 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:33.042953 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" Oct 31 15:48:33.447055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:33.446974 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:48:33.794671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:33.794630 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:34.047403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:34.047322 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:48:34.047873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:34.047833 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:48:34.048453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:34.048435 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:48:38.435383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:38.435332 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:43.134064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:43.134003 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:44.385000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:44.384950 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:44.786598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:44.786563 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:45.419031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:45.418989 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:45.745145 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:45.745108 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:47.491931 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:48:47.491893 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:48:48.170657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:48.170617 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:49.932736 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:49.932699 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:50.128268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:50.128226 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:50.270112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:50.270076 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:50.909199 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:50.909154 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:51.037673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:51.037636 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:53.173618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.173586 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:53.303444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.303409 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:53.426321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.426212 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:48:53.763194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.763158 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq"] Oct 31 15:48:53.770116 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.770082 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:53.776198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.776161 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq"] Oct 31 15:48:53.826811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.826786 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cf95c9\"" Oct 31 15:48:53.826962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.826829 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cf95c9-dockercfg-wj2l5\"" Oct 31 15:48:53.904265 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.904238 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzgl\" (UniqueName: \"kubernetes.io/projected/05e2621d-8790-49d7-8394-f4aaca957c67-kube-api-access-fqzgl\") pod \"payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:53.904420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:53.904364 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e2621d-8790-49d7-8394-f4aaca957c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:54.005542 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:54.005508 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzgl\" (UniqueName: \"kubernetes.io/projected/05e2621d-8790-49d7-8394-f4aaca957c67-kube-api-access-fqzgl\") pod \"payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:54.005706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:54.005603 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e2621d-8790-49d7-8394-f4aaca957c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:54.006298 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:54.006277 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e2621d-8790-49d7-8394-f4aaca957c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:54.063778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:54.063714 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzgl\" (UniqueName: \"kubernetes.io/projected/05e2621d-8790-49d7-8394-f4aaca957c67-kube-api-access-fqzgl\") pod \"payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:54.094341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:54.094316 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:54.475990 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:48:54.475959 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e2621d_8790_49d7_8394_f4aaca957c67.slice/crio-4a0675264acd7416fe4dfc686c3d04222b2046155ee95f0d43df8b71d2649808 WatchSource:0}: Error finding container 4a0675264acd7416fe4dfc686c3d04222b2046155ee95f0d43df8b71d2649808: Status 404 returned error can't find the container with id 4a0675264acd7416fe4dfc686c3d04222b2046155ee95f0d43df8b71d2649808 Oct 31 15:48:54.477680 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:54.477651 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq"] Oct 31 15:48:55.107162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:55.107119 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" event={"ID":"05e2621d-8790-49d7-8394-f4aaca957c67","Type":"ContainerStarted","Data":"934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69"} Oct 31 15:48:55.107363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:55.107209 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" event={"ID":"05e2621d-8790-49d7-8394-f4aaca957c67","Type":"ContainerStarted","Data":"4a0675264acd7416fe4dfc686c3d04222b2046155ee95f0d43df8b71d2649808"} Oct 31 15:48:55.107414 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:55.107358 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:48:55.223656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:55.223603 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" podStartSLOduration=2.223588625 podStartE2EDuration="2.223588625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:55.222623693 +0000 UTC m=+3780.842175105" watchObservedRunningTime="2026-10-31 15:48:55.223588625 +0000 UTC m=+3780.843140032" Oct 31 15:48:58.290831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:48:58.290798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:02.909237 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:02.909201 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:06.118209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:06.118166 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:49:06.559744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:06.559711 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:07.467456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:07.467417 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:07.627713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:07.627677 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:07.745207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:07.745119 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:08.094401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:08.094321 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:12.388682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:12.388641 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:17.089147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:17.089114 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:17.491267 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:49:17.491232 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:49:20.747403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:20.747372 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:20.864195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:20.864161 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:21.722842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:21.722801 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:25.074324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:25.074288 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v"] Oct 31 15:49:25.077617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:25.077594 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd58f104-wkt2v"] Oct 31 15:49:26.431960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:26.431928 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:26.501499 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:26.501446 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd01799a-fe39-4b3c-a57f-612e7c897c33" path="/var/lib/kubelet/pods/cd01799a-fe39-4b3c-a57f-612e7c897c33/volumes" Oct 31 15:49:27.456138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:27.456106 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:49:30.628381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:30.628351 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:31.066793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:31.066762 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:31.518224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:31.518195 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:31.661871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:31.661841 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:31.820085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:31.819989 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:32.415061 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:32.415025 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:32.675225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:32.675135 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:36.573023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:36.572941 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:36.727850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:36.727817 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:37.076060 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:37.076022 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:37.801284 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:37.801247 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:37.952097 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:37.952065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:41.909135 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:41.909099 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:42.033182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.033147 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:42.230005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.229938 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:42.442196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.442164 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp"] Oct 31 15:49:42.446786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.446767 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.501654 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.501585 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp"] Oct 31 15:49:42.505428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.505393 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmxk\" (UniqueName: \"kubernetes.io/projected/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-kube-api-access-ghmxk\") pod \"payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.505616 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.505567 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.562494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.549808 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c5ee32\"" Oct 31 15:49:42.562494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.549977 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c5ee32-dockercfg-s6ggx\"" Oct 31 15:49:42.620893 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.620861 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmxk\" (UniqueName: \"kubernetes.io/projected/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-kube-api-access-ghmxk\") pod \"payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.622165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.622137 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.623541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.623268 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.693170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.693129 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s"] Oct 31 15:49:42.699670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.699625 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s"] Oct 31 15:49:42.699868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.699738 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:42.703395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.703367 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmxk\" (UniqueName: \"kubernetes.io/projected/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-kube-api-access-ghmxk\") pod \"payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.761587 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.761462 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:42.826172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.826014 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/367cacad-404a-4a45-83e2-8ecf95bc9967-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:42.826172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.826094 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xpd\" (UniqueName: \"kubernetes.io/projected/367cacad-404a-4a45-83e2-8ecf95bc9967-kube-api-access-47xpd\") pod \"payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:42.927819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.927783 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/367cacad-404a-4a45-83e2-8ecf95bc9967-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:42.928202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.927839 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xpd\" (UniqueName: \"kubernetes.io/projected/367cacad-404a-4a45-83e2-8ecf95bc9967-kube-api-access-47xpd\") pod \"payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:42.928526 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.928506 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/367cacad-404a-4a45-83e2-8ecf95bc9967-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:42.985951 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:42.985923 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xpd\" (UniqueName: \"kubernetes.io/projected/367cacad-404a-4a45-83e2-8ecf95bc9967-kube-api-access-47xpd\") pod \"payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:43.016519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.016452 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:43.140268 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.140239 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp"] Oct 31 15:49:43.140742 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:49:43.140714 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa6c763_25d8_4f59_b390_8ead05aa3d7f.slice/crio-46a2c22f4df1fe096cb48a8594389477d527e131da8a6d27555898ecedf8ddf0 WatchSource:0}: Error finding container 46a2c22f4df1fe096cb48a8594389477d527e131da8a6d27555898ecedf8ddf0: Status 404 returned error can't find the container with id 46a2c22f4df1fe096cb48a8594389477d527e131da8a6d27555898ecedf8ddf0 Oct 31 15:49:43.395014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.394987 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s"] Oct 31 15:49:43.395262 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:49:43.395241 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367cacad_404a_4a45_83e2_8ecf95bc9967.slice/crio-e5cdf2e37c26d111f9c9e35b5c51ba6833d16dee0257b54a201d0506a0fe4a48 WatchSource:0}: Error finding container e5cdf2e37c26d111f9c9e35b5c51ba6833d16dee0257b54a201d0506a0fe4a48: Status 404 returned error can't find the container with id e5cdf2e37c26d111f9c9e35b5c51ba6833d16dee0257b54a201d0506a0fe4a48 Oct 31 15:49:43.454954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.454924 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" event={"ID":"367cacad-404a-4a45-83e2-8ecf95bc9967","Type":"ContainerStarted","Data":"e5cdf2e37c26d111f9c9e35b5c51ba6833d16dee0257b54a201d0506a0fe4a48"} Oct 31 15:49:43.456866 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.456840 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" event={"ID":"5fa6c763-25d8-4f59-b390-8ead05aa3d7f","Type":"ContainerStarted","Data":"ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0"} Oct 31 15:49:43.456977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.456885 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" event={"ID":"5fa6c763-25d8-4f59-b390-8ead05aa3d7f","Type":"ContainerStarted","Data":"46a2c22f4df1fe096cb48a8594389477d527e131da8a6d27555898ecedf8ddf0"} Oct 31 15:49:43.456977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.456925 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:43.572922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:43.572868 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" podStartSLOduration=1.572852329 podStartE2EDuration="1.572852329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:49:43.571039564 +0000 UTC m=+3829.190590974" watchObservedRunningTime="2026-10-31 15:49:43.572852329 +0000 UTC m=+3829.192403740" Oct 31 15:49:44.464688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:44.464652 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" event={"ID":"367cacad-404a-4a45-83e2-8ecf95bc9967","Type":"ContainerStarted","Data":"7f3e4b2be2799be743c5d52c9719c58566ac480b18b405619f2c71532e707cb1"} Oct 31 15:49:44.465121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:44.464721 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:44.584147 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:44.584084 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" podStartSLOduration=2.584070904 podStartE2EDuration="2.584070904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:49:44.581416781 +0000 UTC m=+3830.200968215" watchObservedRunningTime="2026-10-31 15:49:44.584070904 +0000 UTC m=+3830.203622316" Oct 31 15:49:46.798202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:46.798169 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:47.492915 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:49:47.492877 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:49:51.433573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:51.433536 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:49:54.468098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:54.468062 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:49:55.475286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:55.475259 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:49:56.160930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:49:56.160898 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:00.850709 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:00.850647 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:02.659490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:02.659446 4865 generic.go:356] "Generic (PLEG): container finished" podID="830ca810-fbe2-4e37-a51b-35b66c534776" containerID="d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3" exitCode=0 Oct 31 15:50:02.659979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:02.659528 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" event={"ID":"830ca810-fbe2-4e37-a51b-35b66c534776","Type":"ContainerDied","Data":"d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3"} Oct 31 15:50:02.659979 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:02.659902 4865 scope.go:122] "RemoveContainer" containerID="d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3" Oct 31 15:50:02.661622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:02.661594 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:50:02.661712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:02.661660 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:50:02.661712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:02.661693 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:50:03.669822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:03.669789 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" event={"ID":"830ca810-fbe2-4e37-a51b-35b66c534776","Type":"ContainerStarted","Data":"a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4"} Oct 31 15:50:03.944738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:03.944657 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:50:03.945788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:03.945762 4865 patch_prober.go:29] interesting pod/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Oct 31 15:50:03.945942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:03.945817 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Oct 31 15:50:04.649170 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:04.649142 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:50:04.945216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:04.945135 4865 patch_prober.go:29] interesting pod/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Oct 31 15:50:04.945216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:04.945196 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Oct 31 15:50:05.719513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:05.719435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:05.944480 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:05.944430 4865 patch_prober.go:29] interesting pod/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Oct 31 15:50:05.944671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:05.944534 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Oct 31 15:50:06.936204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:06.936150 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:06.944815 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:06.944787 4865 patch_prober.go:29] interesting pod/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Oct 31 15:50:06.944974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:06.944847 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Oct 31 15:50:07.944745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:07.944709 4865 patch_prober.go:29] interesting pod/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Oct 31 15:50:07.945173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:07.944783 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Oct 31 15:50:08.518987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:08.518944 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:08.944964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:08.944886 4865 patch_prober.go:29] interesting pod/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Oct 31 15:50:08.944964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:08.944956 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Oct 31 15:50:09.949731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:09.949703 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:50:09.950908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:09.950887 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:50:10.420745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:10.420708 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:15.031067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:15.031035 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:17.493131 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:50:17.493086 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:50:19.626742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:19.626697 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:24.307937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:24.307899 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:25.047358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:25.047326 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv"] Oct 31 15:50:25.097443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:25.097416 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0240de5-ll2xv"] Oct 31 15:50:25.749278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:25.749246 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:50:25.981735 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:25.981700 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q"] Oct 31 15:50:25.981942 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:25.981921 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" podUID="f82f2e0a-6663-44c5-be7f-3c22dbec5286" containerName="discovery" containerID="cri-o://237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd" gracePeriod=30 Oct 31 15:50:26.284820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.284794 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:50:26.366353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.366324 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-token\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-token\") pod \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " Oct 31 15:50:26.366552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.366453 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-cacerts\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-cacerts\") pod \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " Oct 31 15:50:26.366552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.366531 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-ca-configmap\") pod \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " Oct 31 15:50:26.366681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.366592 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-dns-cert\") pod \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " Oct 31 15:50:26.366681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.366668 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82f2e0a-6663-44c5-be7f-3c22dbec5286-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82f2e0a-6663-44c5-be7f-3c22dbec5286-local-certs\") pod \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " Oct 31 15:50:26.366813 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.366782 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-kube-api-access-x6snq\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-kube-api-access-x6snq\") pod \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " Oct 31 15:50:26.367001 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.366929 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-kubeconfig\") pod \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\" (UID: \"f82f2e0a-6663-44c5-be7f-3c22dbec5286\") " Oct 31 15:50:26.367239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.367211 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-ca-configmap" pod "f82f2e0a-6663-44c5-be7f-3c22dbec5286" (UID: "f82f2e0a-6663-44c5-be7f-3c22dbec5286"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:50:26.367641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.367615 4865 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-ca-configmap\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:26.369522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.369498 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-token" pod "f82f2e0a-6663-44c5-be7f-3c22dbec5286" (UID: "f82f2e0a-6663-44c5-be7f-3c22dbec5286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:26.369643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.369514 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-dns-cert" pod "f82f2e0a-6663-44c5-be7f-3c22dbec5286" (UID: "f82f2e0a-6663-44c5-be7f-3c22dbec5286"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:50:26.369792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.369766 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-cacerts" pod "f82f2e0a-6663-44c5-be7f-3c22dbec5286" (UID: "f82f2e0a-6663-44c5-be7f-3c22dbec5286"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:50:26.369792 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.369779 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-kubeconfig" pod "f82f2e0a-6663-44c5-be7f-3c22dbec5286" (UID: "f82f2e0a-6663-44c5-be7f-3c22dbec5286"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:50:26.369939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.369784 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82f2e0a-6663-44c5-be7f-3c22dbec5286-local-certs" pod "f82f2e0a-6663-44c5-be7f-3c22dbec5286" (UID: "f82f2e0a-6663-44c5-be7f-3c22dbec5286"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:26.369939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.369928 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-kube-api-access-x6snq" pod "f82f2e0a-6663-44c5-be7f-3c22dbec5286" (UID: "f82f2e0a-6663-44c5-be7f-3c22dbec5286"). InnerVolumeSpecName "kube-api-access-x6snq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:26.468930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.468894 4865 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-cacerts\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:26.468930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.468924 4865 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-csr-dns-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:26.468930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.468934 4865 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82f2e0a-6663-44c5-be7f-3c22dbec5286-local-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:26.469168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.468943 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6snq\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-kube-api-access-x6snq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:26.469168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.468952 4865 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-kubeconfig\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:26.469168 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.468960 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82f2e0a-6663-44c5-be7f-3c22dbec5286-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:26.499126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.499096 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62df08b0-3c14-4c6f-bed5-f416f361d618" path="/var/lib/kubelet/pods/62df08b0-3c14-4c6f-bed5-f416f361d618/volumes" Oct 31 15:50:26.855524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.855395 4865 generic.go:356] "Generic (PLEG): container finished" podID="f82f2e0a-6663-44c5-be7f-3c22dbec5286" containerID="237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd" exitCode=0 Oct 31 15:50:26.855524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.855435 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" event={"ID":"f82f2e0a-6663-44c5-be7f-3c22dbec5286","Type":"ContainerDied","Data":"237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd"} Oct 31 15:50:26.855524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.855458 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" Oct 31 15:50:26.855524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.855498 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q" event={"ID":"f82f2e0a-6663-44c5-be7f-3c22dbec5286","Type":"ContainerDied","Data":"86f35076e3eb73c34e92e37982eb72cc7a56e77c815dc9aa87abf9cee8c63537"} Oct 31 15:50:26.855524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.855524 4865 scope.go:122] "RemoveContainer" containerID="237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd" Oct 31 15:50:26.867822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.867804 4865 scope.go:122] "RemoveContainer" containerID="237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd" Oct 31 15:50:26.868073 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:50:26.868053 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd\": container with ID starting with 237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd not found: ID does not exist" containerID="237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd" Oct 31 15:50:26.868143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:26.868081 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd"} err="failed to get container status \"237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd\": rpc error: code = NotFound desc = could not find container \"237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd\": container with ID starting with 237c4175960f594ab7548dcb0614f90c68aeb42dd22cbbaa3468a9023e2bb0bd not found: ID does not exist" Oct 31 15:50:27.074811 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:27.074781 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q"] Oct 31 15:50:27.078413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:27.078385 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8m82q"] Oct 31 15:50:28.497178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:28.497145 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82f2e0a-6663-44c5-be7f-3c22dbec5286" path="/var/lib/kubelet/pods/f82f2e0a-6663-44c5-be7f-3c22dbec5286/volumes" Oct 31 15:50:29.051411 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:29.051366 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:29.235317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:29.235285 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:29.356738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:29.356654 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:33.791610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:33.791577 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:38.463991 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:38.463953 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:41.452869 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:41.452837 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:43.312332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:43.312296 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:47.020806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:47.020768 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:47.491170 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:50:47.491134 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:50:48.330327 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:48.330275 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:49.215458 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:49.215421 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:52.891251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:52.891218 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:53.020971 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:53.020942 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:54.127175 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:54.127098 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:54.484629 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:54.484590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:54.983633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:54.983602 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:56.124880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:56.124852 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:50:56.134581 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:56.134558 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:50:57.576702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:57.576668 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:57.710314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:57.710233 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:57.842769 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:57.842678 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:50:58.573563 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:58.573541 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:50:58.582025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:58.582003 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:50:59.468347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:59.468309 4865 scope.go:122] "RemoveContainer" containerID="b20c479d329fd741b31260a2aedde2406629479cced86c9706001fbb44c15d3d" Oct 31 15:50:59.499438 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:50:59.499411 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:04.241055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:04.241028 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:07.288784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:07.288705 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:07.408543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:07.408509 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:08.785063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:08.785021 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:08.909603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:08.909573 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:09.300831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:09.300772 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:11.135550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:11.135518 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl"] Oct 31 15:51:11.136019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:11.135701 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" containerID="cri-o://3b7f8582faa8a4a37a62134cd55b8af285709cae999f5f753d6c8514b63a4d5a" gracePeriod=30 Oct 31 15:51:12.679537 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:12.679494 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj"] Oct 31 15:51:13.062659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:13.057746 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:13.191140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:13.191101 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:13.571144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:13.571109 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:13.702573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:13.702541 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:13.904155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:13.904081 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:14.393685 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:14.393650 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:51:14.393891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:14.393719 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:51:15.431747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:15.431697 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:16.287204 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.287172 4865 generic.go:356] "Generic (PLEG): container finished" podID="05802775-dff2-4639-b895-2cbae3b5d52e" containerID="3b7f8582faa8a4a37a62134cd55b8af285709cae999f5f753d6c8514b63a4d5a" exitCode=0 Oct 31 15:51:16.287383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.287247 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" event={"ID":"05802775-dff2-4639-b895-2cbae3b5d52e","Type":"ContainerDied","Data":"3b7f8582faa8a4a37a62134cd55b8af285709cae999f5f753d6c8514b63a4d5a"} Oct 31 15:51:16.454441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.454417 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:51:16.597448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597415 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-data\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.597659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597538 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05802775-dff2-4639-b895-2cbae3b5d52e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05802775-dff2-4639-b895-2cbae3b5d52e-istio-podinfo\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.597659 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597573 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-socket\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.597786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597687 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-credential-socket\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.597786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597737 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-data" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:51:16.597786 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597745 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-envoy\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.597955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597839 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-socket" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:51:16.597955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597882 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-certs\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.597955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597921 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-kube-api-access-lnmm4\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-kube-api-access-lnmm4\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.597955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597949 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05802775-dff2-4639-b895-2cbae3b5d52e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05802775-dff2-4639-b895-2cbae3b5d52e-istiod-ca-cert\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.598176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597979 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-istio-token\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-istio-token\") pod \"05802775-dff2-4639-b895-2cbae3b5d52e\" (UID: \"05802775-dff2-4639-b895-2cbae3b5d52e\") " Oct 31 15:51:16.598176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.597981 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-credential-socket" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:51:16.598176 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.598099 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-certs" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:51:16.598446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.598376 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05802775-dff2-4639-b895-2cbae3b5d52e-istiod-ca-cert" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:51:16.598683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.598663 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.598756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.598685 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05802775-dff2-4639-b895-2cbae3b5d52e-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.598756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.598694 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.598756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.598704 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.598756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.598716 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.600130 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.600100 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05802775-dff2-4639-b895-2cbae3b5d52e-istio-podinfo" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:51:16.600335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.600318 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-istio-token" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:51:16.600636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.600610 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-envoy" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:51:16.600738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.600659 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-kube-api-access-lnmm4" pod "05802775-dff2-4639-b895-2cbae3b5d52e" (UID: "05802775-dff2-4639-b895-2cbae3b5d52e"). InnerVolumeSpecName "kube-api-access-lnmm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:51:16.701052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.700967 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05802775-dff2-4639-b895-2cbae3b5d52e-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.701052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.700998 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05802775-dff2-4639-b895-2cbae3b5d52e-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.701052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.701007 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmm4\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-kube-api-access-lnmm4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:16.701052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:16.701018 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05802775-dff2-4639-b895-2cbae3b5d52e-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:17.295397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:17.295363 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" event={"ID":"05802775-dff2-4639-b895-2cbae3b5d52e","Type":"ContainerDied","Data":"52b40a9f31356e82bed13903c2208b1f4c015b7c2d92d86b710bcf936ef3d92b"} Oct 31 15:51:17.295397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:17.295396 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl" Oct 31 15:51:17.295670 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:17.295417 4865 scope.go:122] "RemoveContainer" containerID="3b7f8582faa8a4a37a62134cd55b8af285709cae999f5f753d6c8514b63a4d5a" Oct 31 15:51:17.496249 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:51:17.496173 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:51:17.516344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:17.516315 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl"] Oct 31 15:51:17.567669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:17.567598 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-kkrtl"] Oct 31 15:51:17.680515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:17.680478 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" podUID="748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" containerName="payload-pre-processing" containerID="cri-o://cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6" gracePeriod=25 Oct 31 15:51:17.961364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:17.961333 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:51:18.120457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.120367 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-kube-api-access-95m2t\" (UniqueName: \"kubernetes.io/projected/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-kube-api-access-95m2t\") pod \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " Oct 31 15:51:18.120457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.120411 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-plugins-config-volume\") pod \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\" (UID: \"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f\") " Oct 31 15:51:18.120852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.120825 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-plugins-config-volume" pod "748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" (UID: "748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:51:18.122801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.122765 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-kube-api-access-95m2t" pod "748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" (UID: "748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f"). InnerVolumeSpecName "kube-api-access-95m2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:51:18.221603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.221575 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:18.221603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.221601 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95m2t\" (UniqueName: \"kubernetes.io/projected/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f-kube-api-access-95m2t\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:18.306159 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.306132 4865 generic.go:356] "Generic (PLEG): container finished" podID="748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" containerID="cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6" exitCode=0 Oct 31 15:51:18.306341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.306178 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" Oct 31 15:51:18.306341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.306196 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" event={"ID":"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f","Type":"ContainerDied","Data":"cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6"} Oct 31 15:51:18.306341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.306243 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj" event={"ID":"748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f","Type":"ContainerDied","Data":"2434379b9a25aeb6fdb6bf7e02cda3b4ada0735b0f6b5ca7151c676e8594cb67"} Oct 31 15:51:18.306341 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.306266 4865 scope.go:122] "RemoveContainer" containerID="cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6" Oct 31 15:51:18.318922 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.318902 4865 scope.go:122] "RemoveContainer" containerID="cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6" Oct 31 15:51:18.319211 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:51:18.319189 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6\": container with ID starting with cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6 not found: ID does not exist" containerID="cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6" Oct 31 15:51:18.319310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.319219 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6"} err="failed to get container status \"cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6\": rpc error: code = NotFound desc = could not find container \"cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6\": container with ID starting with cee6684df9b83672aee36282f6292da6d50b97f6bd31b60ef2d11cc682975de6 not found: ID does not exist" Oct 31 15:51:18.476528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.476497 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj"] Oct 31 15:51:18.479741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.479713 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c4840c-ff659847f-j87nj"] Oct 31 15:51:18.509987 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.509952 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" path="/var/lib/kubelet/pods/05802775-dff2-4639-b895-2cbae3b5d52e/volumes" Oct 31 15:51:18.510519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.510446 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" path="/var/lib/kubelet/pods/748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f/volumes" Oct 31 15:51:18.537952 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.537929 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:18.763615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.763527 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:18.882744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:18.882710 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:19.198066 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:19.197985 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:21.046598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:21.046555 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:23.488753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:23.488723 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:28.574036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:28.573998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:29.062882 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:29.062846 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:51:33.376845 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:33.376790 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:38.015844 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:38.015798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:42.716681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:42.716646 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:47.490172 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:51:47.490129 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:51:47.495189 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:47.495157 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:52.171669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:52.171613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:53.684547 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:53.684512 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:56.846421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:56.846382 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:59.503137 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.503105 4865 scope.go:122] "RemoveContainer" containerID="cd7bcc8cbfea4fe2a7d64bdd2b96794774ffacf3a8513a54de70abc17f8fc26a" Oct 31 15:51:59.709562 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.709529 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:51:59.888886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.888805 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh"] Oct 31 15:51:59.889571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889546 4865 cpu_manager.go:414] "containerMap: removing container" podUID="748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" containerName="payload-pre-processing" Oct 31 15:51:59.889714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889569 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" containerName="payload-pre-processing" Oct 31 15:51:59.889714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889604 4865 cpu_manager.go:414] "containerMap: removing container" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" Oct 31 15:51:59.889714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889612 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" Oct 31 15:51:59.889714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889664 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f82f2e0a-6663-44c5-be7f-3c22dbec5286" containerName="discovery" Oct 31 15:51:59.889714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889670 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82f2e0a-6663-44c5-be7f-3c22dbec5286" containerName="discovery" Oct 31 15:51:59.889912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889788 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="748e3dee-d8d1-4fe0-ae01-4d1f8b87fd5f" containerName="payload-pre-processing" Oct 31 15:51:59.889912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889800 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="05802775-dff2-4639-b895-2cbae3b5d52e" containerName="istio-proxy" Oct 31 15:51:59.889912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.889812 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82f2e0a-6663-44c5-be7f-3c22dbec5286" containerName="discovery" Oct 31 15:51:59.894934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.894917 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915491 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mnzf\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-kube-api-access-9mnzf\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915541 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915640 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915754 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915814 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915792 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915846 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915885 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915906 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.915973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.915937 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:51:59.994335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:51:59.994296 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh"] Oct 31 15:52:00.016742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.016712 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mnzf\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-kube-api-access-9mnzf\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.016941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.016763 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.016941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.016815 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.016941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.016903 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.017141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.016956 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.017753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.017684 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.017753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.017764 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.017753 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.017784 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.018071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.017821 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.018071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.017875 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.018071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.018049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.018306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.018261 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.018420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.018399 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.018565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.018548 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.019602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.019574 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.020055 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.020033 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.077121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.077084 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.077293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.077278 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mnzf\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-kube-api-access-9mnzf\") pod \"maas-default-gateway-openshift-default-67bf986f99-zmprh\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.209575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.209545 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:00.639574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.639546 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh"] Oct 31 15:52:00.640136 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:52:00.640106 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eecba10_2f62_43a2_9ee0_f6ffb8973333.slice/crio-819e407f8d93789377d1bd88bcaaa692945c8eb71eec47bb2e903ae0267bafa4 WatchSource:0}: Error finding container 819e407f8d93789377d1bd88bcaaa692945c8eb71eec47bb2e903ae0267bafa4: Status 404 returned error can't find the container with id 819e407f8d93789377d1bd88bcaaa692945c8eb71eec47bb2e903ae0267bafa4 Oct 31 15:52:00.643258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.643224 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:52:00.643339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.643293 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:52:00.643339 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.643322 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:52:00.669214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.669182 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" event={"ID":"0eecba10-2f62-43a2-9ee0-f6ffb8973333","Type":"ContainerStarted","Data":"819e407f8d93789377d1bd88bcaaa692945c8eb71eec47bb2e903ae0267bafa4"} Oct 31 15:52:00.813644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:00.813608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:01.044703 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:01.044655 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:01.679058 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:01.679020 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" event={"ID":"0eecba10-2f62-43a2-9ee0-f6ffb8973333","Type":"ContainerStarted","Data":"8fb5bcbe22f079755b309de96787c0d3f3cafc52c26cccc84a0a86070a455348"} Oct 31 15:52:01.680860 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:01.680834 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:01.850599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:01.850296 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podStartSLOduration=2.850277802 podStartE2EDuration="2.850277802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:52:01.84788954 +0000 UTC m=+3967.467440952" watchObservedRunningTime="2026-10-31 15:52:01.850277802 +0000 UTC m=+3967.469829214" Oct 31 15:52:02.215557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:02.215523 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:02.215998 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:02.215963 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 15:52:02.216182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:02.216157 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 15:52:03.210871 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:03.210832 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 15:52:03.211529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:03.211461 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 15:52:04.211787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:04.211748 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 15:52:04.212441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:04.212411 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 15:52:05.149132 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:05.149095 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:05.212495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:05.210462 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 15:52:05.212495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:05.210564 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 15:52:06.214899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:06.214860 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 15:52:06.215552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:06.215515 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 15:52:06.720393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:06.720359 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:07.210082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:07.210053 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 15:52:07.210263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:07.210115 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 15:52:07.629152 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:07.629070 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:08.210366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:08.210329 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Oct 31 15:52:08.210571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:08.210396 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Oct 31 15:52:09.123824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:09.123782 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:10.100227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:10.100188 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:10.100789 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:10.100764 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:10.101768 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:10.101750 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 15:52:11.311527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:11.311490 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:16.008527 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:16.008463 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:17.492615 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:52:17.492578 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:52:20.942306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:20.942265 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:22.118803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.118766 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd"] Oct 31 15:52:22.123328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.123310 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.148164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.148139 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.148289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.148177 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.148289 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.148196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v449c\" (UniqueName: \"kubernetes.io/projected/2f82d107-968b-4d89-9f9f-55051c595f80-kube-api-access-v449c\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.179033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.179002 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd"] Oct 31 15:52:22.225712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.225674 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:52:22.225712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.225674 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:52:22.225918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.225684 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:52:22.230854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.230835 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:52:22.249207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.249165 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.249398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.249219 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.249398 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.249240 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v449c\" (UniqueName: \"kubernetes.io/projected/2f82d107-968b-4d89-9f9f-55051c595f80-kube-api-access-v449c\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.250104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.250078 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.252095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.252075 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.307738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.307710 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v449c\" (UniqueName: \"kubernetes.io/projected/2f82d107-968b-4d89-9f9f-55051c595f80-kube-api-access-v449c\") pod \"maas-api-revoke-sub-db0ef4ab-52xhd\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.438912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.438830 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:22.818939 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:52:22.818907 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f82d107_968b_4d89_9f9f_55051c595f80.slice/crio-91ef7c355cf974ebadd3d81af45ce7bcae77ea70d4feee9e058a587940bdd40d WatchSource:0}: Error finding container 91ef7c355cf974ebadd3d81af45ce7bcae77ea70d4feee9e058a587940bdd40d: Status 404 returned error can't find the container with id 91ef7c355cf974ebadd3d81af45ce7bcae77ea70d4feee9e058a587940bdd40d Oct 31 15:52:22.818939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.818926 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd"] Oct 31 15:52:22.861682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:22.861653 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" event={"ID":"2f82d107-968b-4d89-9f9f-55051c595f80","Type":"ContainerStarted","Data":"91ef7c355cf974ebadd3d81af45ce7bcae77ea70d4feee9e058a587940bdd40d"} Oct 31 15:52:23.871162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:23.871122 4865 generic.go:356] "Generic (PLEG): container finished" podID="2f82d107-968b-4d89-9f9f-55051c595f80" containerID="e53ef43a4b4a5075fe32a0944e6b2081989fe00929f505ea585bb62b7063ecbc" exitCode=0 Oct 31 15:52:23.871668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:23.871363 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" event={"ID":"2f82d107-968b-4d89-9f9f-55051c595f80","Type":"ContainerDied","Data":"e53ef43a4b4a5075fe32a0944e6b2081989fe00929f505ea585bb62b7063ecbc"} Oct 31 15:52:25.042775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.042745 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:25.103573 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.103539 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f82d107-968b-4d89-9f9f-55051c595f80-kube-api-access-v449c\" (UniqueName: \"kubernetes.io/projected/2f82d107-968b-4d89-9f9f-55051c595f80-kube-api-access-v449c\") pod \"2f82d107-968b-4d89-9f9f-55051c595f80\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " Oct 31 15:52:25.103756 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.103644 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-xks-ca\") pod \"2f82d107-968b-4d89-9f9f-55051c595f80\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " Oct 31 15:52:25.103835 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.103818 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-openshift-ca\") pod \"2f82d107-968b-4d89-9f9f-55051c595f80\" (UID: \"2f82d107-968b-4d89-9f9f-55051c595f80\") " Oct 31 15:52:25.104155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.104130 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-openshift-ca" pod "2f82d107-968b-4d89-9f9f-55051c595f80" (UID: "2f82d107-968b-4d89-9f9f-55051c595f80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:52:25.104441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.104386 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:25.105880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.105855 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-xks-ca" pod "2f82d107-968b-4d89-9f9f-55051c595f80" (UID: "2f82d107-968b-4d89-9f9f-55051c595f80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:52:25.106003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.105932 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f82d107-968b-4d89-9f9f-55051c595f80-kube-api-access-v449c" pod "2f82d107-968b-4d89-9f9f-55051c595f80" (UID: "2f82d107-968b-4d89-9f9f-55051c595f80"). InnerVolumeSpecName "kube-api-access-v449c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:52:25.206115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.206032 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v449c\" (UniqueName: \"kubernetes.io/projected/2f82d107-968b-4d89-9f9f-55051c595f80-kube-api-access-v449c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:25.206115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.206064 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f82d107-968b-4d89-9f9f-55051c595f80-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:25.623000 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.622968 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:25.886575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.886486 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" Oct 31 15:52:25.886575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.886504 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd" event={"ID":"2f82d107-968b-4d89-9f9f-55051c595f80","Type":"ContainerDied","Data":"91ef7c355cf974ebadd3d81af45ce7bcae77ea70d4feee9e058a587940bdd40d"} Oct 31 15:52:25.886575 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:25.886556 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91ef7c355cf974ebadd3d81af45ce7bcae77ea70d4feee9e058a587940bdd40d" Oct 31 15:52:30.309365 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:30.309311 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:32.998111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:32.998080 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:34.397497 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:34.397434 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:36.103621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:36.103539 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:37.965684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:37.965652 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4"] Oct 31 15:52:37.966424 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:37.966402 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2f82d107-968b-4d89-9f9f-55051c595f80" containerName="revoke-keys" Oct 31 15:52:37.966551 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:37.966424 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f82d107-968b-4d89-9f9f-55051c595f80" containerName="revoke-keys" Oct 31 15:52:37.966664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:37.966652 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f82d107-968b-4d89-9f9f-55051c595f80" containerName="revoke-keys" Oct 31 15:52:37.970506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:37.970460 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.024997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.024964 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4"] Oct 31 15:52:38.072011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.071975 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:52:38.072011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.071974 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:52:38.072279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.072025 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:52:38.072279 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.072091 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:52:38.087164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.087128 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fk84\" (UniqueName: \"kubernetes.io/projected/f545b0b7-1395-4c93-a084-9e8c2b58b759-kube-api-access-7fk84\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.087311 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.087193 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.087380 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.087360 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.188714 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.188679 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fk84\" (UniqueName: \"kubernetes.io/projected/f545b0b7-1395-4c93-a084-9e8c2b58b759-kube-api-access-7fk84\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.188916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.188736 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.188916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.188811 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.189548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.189523 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.191392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.191359 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.246790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.246722 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fk84\" (UniqueName: \"kubernetes.io/projected/f545b0b7-1395-4c93-a084-9e8c2b58b759-kube-api-access-7fk84\") pod \"maas-api-revoke-keys-e2e-inf-a-c4840c-625h4\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.284028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.284000 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:38.617799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.617707 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:38.664841 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:52:38.664798 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf545b0b7_1395_4c93_a084_9e8c2b58b759.slice/crio-3913e9890a009e174956ac9ab8c01a6d6c727b36282c233812cbedc0a04d5c59 WatchSource:0}: Error finding container 3913e9890a009e174956ac9ab8c01a6d6c727b36282c233812cbedc0a04d5c59: Status 404 returned error can't find the container with id 3913e9890a009e174956ac9ab8c01a6d6c727b36282c233812cbedc0a04d5c59 Oct 31 15:52:38.672495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.667883 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4"] Oct 31 15:52:38.976767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:38.976732 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" event={"ID":"f545b0b7-1395-4c93-a084-9e8c2b58b759","Type":"ContainerStarted","Data":"3913e9890a009e174956ac9ab8c01a6d6c727b36282c233812cbedc0a04d5c59"} Oct 31 15:52:39.984874 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:39.984834 4865 generic.go:356] "Generic (PLEG): container finished" podID="f545b0b7-1395-4c93-a084-9e8c2b58b759" containerID="11d7c4b499d49d3b90bc44e2b222d174c49dda73a45fe63d0829899b0a57f34f" exitCode=0 Oct 31 15:52:39.985303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:39.984923 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" event={"ID":"f545b0b7-1395-4c93-a084-9e8c2b58b759","Type":"ContainerDied","Data":"11d7c4b499d49d3b90bc44e2b222d174c49dda73a45fe63d0829899b0a57f34f"} Oct 31 15:52:40.784350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:40.784315 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:40.910758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:40.910727 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:41.159403 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.159372 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:41.185787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.185744 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:41.231101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.231042 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f545b0b7-1395-4c93-a084-9e8c2b58b759-kube-api-access-7fk84\" (UniqueName: \"kubernetes.io/projected/f545b0b7-1395-4c93-a084-9e8c2b58b759-kube-api-access-7fk84\") pod \"f545b0b7-1395-4c93-a084-9e8c2b58b759\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " Oct 31 15:52:41.232242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.231342 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-xks-ca\") pod \"f545b0b7-1395-4c93-a084-9e8c2b58b759\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " Oct 31 15:52:41.232242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.231451 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-openshift-ca\") pod \"f545b0b7-1395-4c93-a084-9e8c2b58b759\" (UID: \"f545b0b7-1395-4c93-a084-9e8c2b58b759\") " Oct 31 15:52:41.232446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.232373 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-openshift-ca" pod "f545b0b7-1395-4c93-a084-9e8c2b58b759" (UID: "f545b0b7-1395-4c93-a084-9e8c2b58b759"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:52:41.238593 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.238564 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f545b0b7-1395-4c93-a084-9e8c2b58b759-kube-api-access-7fk84" pod "f545b0b7-1395-4c93-a084-9e8c2b58b759" (UID: "f545b0b7-1395-4c93-a084-9e8c2b58b759"). InnerVolumeSpecName "kube-api-access-7fk84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:52:41.250688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.250630 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-xks-ca" pod "f545b0b7-1395-4c93-a084-9e8c2b58b759" (UID: "f545b0b7-1395-4c93-a084-9e8c2b58b759"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:52:41.333215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.333133 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:41.333215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.333167 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fk84\" (UniqueName: \"kubernetes.io/projected/f545b0b7-1395-4c93-a084-9e8c2b58b759-kube-api-access-7fk84\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:41.333215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.333182 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f545b0b7-1395-4c93-a084-9e8c2b58b759-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:41.415775 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.415738 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj"] Oct 31 15:52:41.416040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.416010 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" containerID="cri-o://41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987" gracePeriod=30 Oct 31 15:52:41.442603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:41.442572 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:42.000816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:42.000792 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" Oct 31 15:52:42.000977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:42.000829 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4" event={"ID":"f545b0b7-1395-4c93-a084-9e8c2b58b759","Type":"ContainerDied","Data":"3913e9890a009e174956ac9ab8c01a6d6c727b36282c233812cbedc0a04d5c59"} Oct 31 15:52:42.000977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:42.000877 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3913e9890a009e174956ac9ab8c01a6d6c727b36282c233812cbedc0a04d5c59" Oct 31 15:52:42.172425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:42.172367 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:44.317590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:44.317556 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:44.454895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:44.454857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:46.194704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.194670 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:46.696601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.696533 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:52:46.818974 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.818940 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-certs\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.818991 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-kube-api-access-8qvfl\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-kube-api-access-8qvfl\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819095 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-envoy\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819125 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-credential-socket\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819190 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-socket\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819223 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-certs" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:52:46.819392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819305 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-data\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819392 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819374 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a507b21c-dc8b-4741-a732-a5a3cb96f161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a507b21c-dc8b-4741-a732-a5a3cb96f161-istiod-ca-cert\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819430 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-podinfo\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819460 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-credential-socket" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:52:46.819559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819524 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-socket" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:52:46.819746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819612 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-data" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:52:46.819746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819634 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-token\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-token\") pod \"a507b21c-dc8b-4741-a732-a5a3cb96f161\" (UID: \"a507b21c-dc8b-4741-a732-a5a3cb96f161\") " Oct 31 15:52:46.819864 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.819834 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a507b21c-dc8b-4741-a732-a5a3cb96f161-istiod-ca-cert" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:52:46.820505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.820460 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.820634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.820507 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a507b21c-dc8b-4741-a732-a5a3cb96f161-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.820634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.820522 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.820634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.820536 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.820634 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.820549 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.821433 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.821407 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-envoy" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:52:46.821645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.821625 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-kube-api-access-8qvfl" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "kube-api-access-8qvfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:52:46.821730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.821653 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-podinfo" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:52:46.821940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.821918 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-token" pod "a507b21c-dc8b-4741-a732-a5a3cb96f161" (UID: "a507b21c-dc8b-4741-a732-a5a3cb96f161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:52:46.921658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.921629 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.921658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.921659 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.921930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.921676 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvfl\" (UniqueName: \"kubernetes.io/projected/a507b21c-dc8b-4741-a732-a5a3cb96f161-kube-api-access-8qvfl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:46.921930 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:46.921691 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a507b21c-dc8b-4741-a732-a5a3cb96f161-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:47.035335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.035296 4865 generic.go:356] "Generic (PLEG): container finished" podID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerID="41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987" exitCode=0 Oct 31 15:52:47.035594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.035354 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" Oct 31 15:52:47.035594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.035377 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" event={"ID":"a507b21c-dc8b-4741-a732-a5a3cb96f161","Type":"ContainerDied","Data":"41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987"} Oct 31 15:52:47.035594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.035434 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj" event={"ID":"a507b21c-dc8b-4741-a732-a5a3cb96f161","Type":"ContainerDied","Data":"c633be47406e3a4355578f86fc5899da337155d1baf70cdd81326351fb4f8010"} Oct 31 15:52:47.035594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.035457 4865 scope.go:122] "RemoveContainer" containerID="41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987" Oct 31 15:52:47.048067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.048049 4865 scope.go:122] "RemoveContainer" containerID="eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723" Oct 31 15:52:47.055726 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.055696 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:47.059054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.059034 4865 scope.go:122] "RemoveContainer" containerID="41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987" Oct 31 15:52:47.059360 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:52:47.059339 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987\": container with ID starting with 41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987 not found: ID does not exist" containerID="41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987" Oct 31 15:52:47.059417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.059371 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987"} err="failed to get container status \"41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987\": rpc error: code = NotFound desc = could not find container \"41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987\": container with ID starting with 41de6b7c8f25061277a95b9b04f942d686e686669cc24961969cae21d6059987 not found: ID does not exist" Oct 31 15:52:47.059417 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.059390 4865 scope.go:122] "RemoveContainer" containerID="eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723" Oct 31 15:52:47.059720 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:52:47.059700 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723\": container with ID starting with eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723 not found: ID does not exist" containerID="eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723" Oct 31 15:52:47.059770 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.059725 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723"} err="failed to get container status \"eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723\": rpc error: code = NotFound desc = could not find container \"eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723\": container with ID starting with eb0ac087305c8595b1b09ba59f0bf0815fdb74a002753f66474eb0840dfc0723 not found: ID does not exist" Oct 31 15:52:47.203610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.203570 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj"] Oct 31 15:52:47.208909 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.208880 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b18df8-openshift-default-6464676cf7-bjrbj"] Oct 31 15:52:47.492269 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:52:47.492235 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:52:47.761353 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.761311 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:47.921669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:47.921608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:48.159033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:48.158948 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:48.497410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:48.497376 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" path="/var/lib/kubelet/pods/a507b21c-dc8b-4741-a732-a5a3cb96f161/volumes" Oct 31 15:52:49.897778 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:49.897721 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq"] Oct 31 15:52:50.131242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:50.131201 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:50.261742 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:50.261693 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:50.715507 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:50.715455 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:52.547218 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:52.547171 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:52.666999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:52.666957 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:53.886049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:53.886011 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:54.498354 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:54.498320 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:52:54.899194 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:54.899098 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" podUID="504e7fdf-388b-4db6-951f-376b2117bd22" containerName="payload-pre-processing" containerID="cri-o://9dd2710b5fa910813d47af42beb1fda24f6b016cc12b5a05092d4129558453b1" gracePeriod=25 Oct 31 15:52:55.102331 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.102296 4865 generic.go:356] "Generic (PLEG): container finished" podID="504e7fdf-388b-4db6-951f-376b2117bd22" containerID="9dd2710b5fa910813d47af42beb1fda24f6b016cc12b5a05092d4129558453b1" exitCode=0 Oct 31 15:52:55.102540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.102355 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" event={"ID":"504e7fdf-388b-4db6-951f-376b2117bd22","Type":"ContainerDied","Data":"9dd2710b5fa910813d47af42beb1fda24f6b016cc12b5a05092d4129558453b1"} Oct 31 15:52:55.170827 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.170756 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:52:55.248843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.248813 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504e7fdf-388b-4db6-951f-376b2117bd22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e7fdf-388b-4db6-951f-376b2117bd22-plugins-config-volume\") pod \"504e7fdf-388b-4db6-951f-376b2117bd22\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " Oct 31 15:52:55.249005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.248963 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504e7fdf-388b-4db6-951f-376b2117bd22-kube-api-access-ht66c\" (UniqueName: \"kubernetes.io/projected/504e7fdf-388b-4db6-951f-376b2117bd22-kube-api-access-ht66c\") pod \"504e7fdf-388b-4db6-951f-376b2117bd22\" (UID: \"504e7fdf-388b-4db6-951f-376b2117bd22\") " Oct 31 15:52:55.249184 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.249155 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504e7fdf-388b-4db6-951f-376b2117bd22-plugins-config-volume" pod "504e7fdf-388b-4db6-951f-376b2117bd22" (UID: "504e7fdf-388b-4db6-951f-376b2117bd22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:52:55.249343 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.249327 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504e7fdf-388b-4db6-951f-376b2117bd22-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:55.251235 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.251215 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504e7fdf-388b-4db6-951f-376b2117bd22-kube-api-access-ht66c" pod "504e7fdf-388b-4db6-951f-376b2117bd22" (UID: "504e7fdf-388b-4db6-951f-376b2117bd22"). InnerVolumeSpecName "kube-api-access-ht66c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:52:55.350598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:55.350564 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht66c\" (UniqueName: \"kubernetes.io/projected/504e7fdf-388b-4db6-951f-376b2117bd22-kube-api-access-ht66c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:56.148485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:56.148427 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" event={"ID":"504e7fdf-388b-4db6-951f-376b2117bd22","Type":"ContainerDied","Data":"ca3a667487282c1cb3b33121b8e7efec4a5d52c5bcda32fbe84abec13e7c3a60"} Oct 31 15:52:56.149261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:56.149238 4865 scope.go:122] "RemoveContainer" containerID="9dd2710b5fa910813d47af42beb1fda24f6b016cc12b5a05092d4129558453b1" Oct 31 15:52:56.149557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:56.149176 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq" Oct 31 15:52:56.331492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:56.331157 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq"] Oct 31 15:52:56.377765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:56.377733 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c9a78-54cb8dd4f5-7v5cq"] Oct 31 15:52:56.547439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:56.547399 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504e7fdf-388b-4db6-951f-376b2117bd22" path="/var/lib/kubelet/pods/504e7fdf-388b-4db6-951f-376b2117bd22/volumes" Oct 31 15:52:57.747494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:57.745106 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:59.438493 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:59.435900 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:52:59.570503 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:52:59.569856 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:00.957342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:00.957280 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:01.302128 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:01.302038 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:01.439948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:01.439901 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:01.451404 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:01.451380 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc"] Oct 31 15:53:01.451655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:01.451636 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" containerID="cri-o://a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4" gracePeriod=30 Oct 31 15:53:02.383900 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:02.383859 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:02.521406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:02.521371 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:02.657595 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:02.657516 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:06.734107 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.734048 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:53:06.768533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768463 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/830ca810-fbe2-4e37-a51b-35b66c534776-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830ca810-fbe2-4e37-a51b-35b66c534776-istio-podinfo\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.768710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768557 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-certs\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.768710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768630 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-credential-socket\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.768710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768680 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-istio-token\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-istio-token\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.768710 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768706 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830ca810-fbe2-4e37-a51b-35b66c534776-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830ca810-fbe2-4e37-a51b-35b66c534776-istiod-ca-cert\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.768949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768739 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-envoy\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.768949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768766 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-socket\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.768949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768893 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-certs" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:53:06.769113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.768967 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-data\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.769113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769009 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-kube-api-access-5pwq2\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-kube-api-access-5pwq2\") pod \"830ca810-fbe2-4e37-a51b-35b66c534776\" (UID: \"830ca810-fbe2-4e37-a51b-35b66c534776\") " Oct 31 15:53:06.769113 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769002 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-credential-socket" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:53:06.769292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769144 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-socket" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:53:06.769292 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769155 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830ca810-fbe2-4e37-a51b-35b66c534776-istiod-ca-cert" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:53:06.769519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769464 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-data" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:53:06.769826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769801 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.769907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769827 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.769907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769842 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.769907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769855 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830ca810-fbe2-4e37-a51b-35b66c534776-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.769907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.769868 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.771321 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.771295 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-envoy" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:53:06.771436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.771321 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/830ca810-fbe2-4e37-a51b-35b66c534776-istio-podinfo" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:53:06.771889 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.771873 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-kube-api-access-5pwq2" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "kube-api-access-5pwq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:53:06.771960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.771891 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-istio-token" pod "830ca810-fbe2-4e37-a51b-35b66c534776" (UID: "830ca810-fbe2-4e37-a51b-35b66c534776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:53:06.871426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.871390 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwq2\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-kube-api-access-5pwq2\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.871426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.871424 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830ca810-fbe2-4e37-a51b-35b66c534776-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.871426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.871435 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830ca810-fbe2-4e37-a51b-35b66c534776-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:06.871695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:06.871445 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830ca810-fbe2-4e37-a51b-35b66c534776-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:07.305485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.305431 4865 generic.go:356] "Generic (PLEG): container finished" podID="830ca810-fbe2-4e37-a51b-35b66c534776" containerID="a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4" exitCode=0 Oct 31 15:53:07.305689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.305506 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" event={"ID":"830ca810-fbe2-4e37-a51b-35b66c534776","Type":"ContainerDied","Data":"a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4"} Oct 31 15:53:07.305689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.305574 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" event={"ID":"830ca810-fbe2-4e37-a51b-35b66c534776","Type":"ContainerDied","Data":"4f75aabce7a664f78ed48ba6759a0187e74a81437161a9e12a7c63504b5bcb29"} Oct 31 15:53:07.305689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.305519 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc" Oct 31 15:53:07.305689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.305608 4865 scope.go:122] "RemoveContainer" containerID="a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4" Oct 31 15:53:07.317891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.317872 4865 scope.go:122] "RemoveContainer" containerID="d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3" Oct 31 15:53:07.327560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.327537 4865 scope.go:122] "RemoveContainer" containerID="a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4" Oct 31 15:53:07.327823 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:53:07.327802 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4\": container with ID starting with a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4 not found: ID does not exist" containerID="a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4" Oct 31 15:53:07.327915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.327834 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4"} err="failed to get container status \"a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4\": rpc error: code = NotFound desc = could not find container \"a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4\": container with ID starting with a134d76bbc4efd61c62d6b4ef936a6be03892791d146141a685f740ee484a8f4 not found: ID does not exist" Oct 31 15:53:07.327915 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.327859 4865 scope.go:122] "RemoveContainer" containerID="d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3" Oct 31 15:53:07.328117 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:53:07.328100 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3\": container with ID starting with d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3 not found: ID does not exist" containerID="d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3" Oct 31 15:53:07.328163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.328123 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3"} err="failed to get container status \"d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3\": rpc error: code = NotFound desc = could not find container \"d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3\": container with ID starting with d718ed953f3234c5d6bad5fe3cb94d686d74db5ea82d923030cd708a0f5847d3 not found: ID does not exist" Oct 31 15:53:07.473713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.473676 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc"] Oct 31 15:53:07.525062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.525026 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4c9a78-openshift-default-869b64f5b5-5t8zc"] Oct 31 15:53:07.693281 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.693197 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:07.981319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:07.981281 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:08.498397 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:08.498362 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" path="/var/lib/kubelet/pods/830ca810-fbe2-4e37-a51b-35b66c534776/volumes" Oct 31 15:53:09.467336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:09.467295 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:12.580102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:12.580060 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:17.303541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:17.303463 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:17.491006 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:53:17.490973 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:53:18.897646 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:18.897612 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:22.025102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:22.025064 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:24.334430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.334390 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58"] Oct 31 15:53:24.335247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335223 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f545b0b7-1395-4c93-a084-9e8c2b58b759" containerName="revoke-keys" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335249 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f545b0b7-1395-4c93-a084-9e8c2b58b759" containerName="revoke-keys" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335261 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335269 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335282 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335290 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335350 4865 cpu_manager.go:414] "containerMap: removing container" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335358 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335368 4865 cpu_manager.go:414] "containerMap: removing container" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" Oct 31 15:53:24.335378 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335374 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335387 4865 cpu_manager.go:414] "containerMap: removing container" podUID="504e7fdf-388b-4db6-951f-376b2117bd22" containerName="payload-pre-processing" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335394 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504e7fdf-388b-4db6-951f-376b2117bd22" containerName="payload-pre-processing" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335579 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335596 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="830ca810-fbe2-4e37-a51b-35b66c534776" containerName="istio-proxy" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335605 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="504e7fdf-388b-4db6-951f-376b2117bd22" containerName="payload-pre-processing" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335624 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335634 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a507b21c-dc8b-4741-a732-a5a3cb96f161" containerName="istio-proxy" Oct 31 15:53:24.335905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.335643 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f545b0b7-1395-4c93-a084-9e8c2b58b759" containerName="revoke-keys" Oct 31 15:53:24.339335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.339314 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.395100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.395075 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58"] Oct 31 15:53:24.397215 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.397180 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wvq\" (UniqueName: \"kubernetes.io/projected/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-kube-api-access-s7wvq\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.397372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.397347 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.397519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.397501 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.440777 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.440749 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:53:24.440931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.440912 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:53:24.440977 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.440915 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:53:24.443069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.443053 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:53:24.499153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.499108 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.499313 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.499244 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wvq\" (UniqueName: \"kubernetes.io/projected/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-kube-api-access-s7wvq\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.499382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.499363 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.500035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.500010 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.504018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.503985 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.558063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.558029 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wvq\" (UniqueName: \"kubernetes.io/projected/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-kube-api-access-s7wvq\") pod \"maas-api-revoke-keys-e2e-inf-a-883697-ghn58\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.653062 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.652997 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:24.722818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:24.722716 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:25.029917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:25.029888 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58"] Oct 31 15:53:25.030215 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:53:25.030191 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36cb0c0_0bfb_4765_9959_44e4f514ee8d.slice/crio-4a4f38ab57ee5c7b165a7d665d2ed3a2e927d9f439b451750f18d27f03a84772 WatchSource:0}: Error finding container 4a4f38ab57ee5c7b165a7d665d2ed3a2e927d9f439b451750f18d27f03a84772: Status 404 returned error can't find the container with id 4a4f38ab57ee5c7b165a7d665d2ed3a2e927d9f439b451750f18d27f03a84772 Oct 31 15:53:25.431993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:25.431956 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" event={"ID":"f36cb0c0-0bfb-4765-9959-44e4f514ee8d","Type":"ContainerStarted","Data":"01443565ba77f82b2a877d3b81e59f009a24d7c360f45a40e6d4d4e348d83959"} Oct 31 15:53:25.432340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:25.432023 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" event={"ID":"f36cb0c0-0bfb-4765-9959-44e4f514ee8d","Type":"ContainerStarted","Data":"4a4f38ab57ee5c7b165a7d665d2ed3a2e927d9f439b451750f18d27f03a84772"} Oct 31 15:53:25.571016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:25.570925 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" podStartSLOduration=1.570909648 podStartE2EDuration="1.570909648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:53:25.568389577 +0000 UTC m=+4051.187940988" watchObservedRunningTime="2026-10-31 15:53:25.570909648 +0000 UTC m=+4051.190461058" Oct 31 15:53:26.440586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:26.440501 4865 generic.go:356] "Generic (PLEG): container finished" podID="f36cb0c0-0bfb-4765-9959-44e4f514ee8d" containerID="01443565ba77f82b2a877d3b81e59f009a24d7c360f45a40e6d4d4e348d83959" exitCode=0 Oct 31 15:53:26.440984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:26.440585 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" event={"ID":"f36cb0c0-0bfb-4765-9959-44e4f514ee8d","Type":"ContainerDied","Data":"01443565ba77f82b2a877d3b81e59f009a24d7c360f45a40e6d4d4e348d83959"} Oct 31 15:53:26.986076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:26.986038 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:27.609598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.609574 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:27.641746 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.641718 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-openshift-ca\") pod \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " Oct 31 15:53:27.641944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.641818 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-kube-api-access-s7wvq\" (UniqueName: \"kubernetes.io/projected/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-kube-api-access-s7wvq\") pod \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " Oct 31 15:53:27.641944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.641847 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-xks-ca\") pod \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\" (UID: \"f36cb0c0-0bfb-4765-9959-44e4f514ee8d\") " Oct 31 15:53:27.642160 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.642134 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-openshift-ca" pod "f36cb0c0-0bfb-4765-9959-44e4f514ee8d" (UID: "f36cb0c0-0bfb-4765-9959-44e4f514ee8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:53:27.642363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.642346 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:27.644252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.644223 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-kube-api-access-s7wvq" pod "f36cb0c0-0bfb-4765-9959-44e4f514ee8d" (UID: "f36cb0c0-0bfb-4765-9959-44e4f514ee8d"). InnerVolumeSpecName "kube-api-access-s7wvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:53:27.644490 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.644450 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-xks-ca" pod "f36cb0c0-0bfb-4765-9959-44e4f514ee8d" (UID: "f36cb0c0-0bfb-4765-9959-44e4f514ee8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:53:27.743552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.743520 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wvq\" (UniqueName: \"kubernetes.io/projected/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-kube-api-access-s7wvq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:27.743552 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:27.743548 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36cb0c0-0bfb-4765-9959-44e4f514ee8d-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:53:28.454657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:28.454625 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" Oct 31 15:53:28.454657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:28.454644 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58" event={"ID":"f36cb0c0-0bfb-4765-9959-44e4f514ee8d","Type":"ContainerDied","Data":"4a4f38ab57ee5c7b165a7d665d2ed3a2e927d9f439b451750f18d27f03a84772"} Oct 31 15:53:28.454898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:28.454694 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a4f38ab57ee5c7b165a7d665d2ed3a2e927d9f439b451750f18d27f03a84772" Oct 31 15:53:31.808954 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:31.808910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:36.594372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:36.594335 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:41.313389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:41.313355 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:44.963389 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:44.963353 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:45.079139 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:45.079105 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:53:45.971162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:45.971126 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:47.490618 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:53:47.490580 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:53:50.612596 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:50.612558 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:55.322529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:55.322491 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:53:59.364261 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:53:59.364214 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:00.827401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:00.827365 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:00.949227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:00.949193 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:01.073716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:01.073683 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:03.468973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:03.468936 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:03.595980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:03.595952 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:03.766917 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:03.766834 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:04.403693 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:04.403651 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:05.181849 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:05.181818 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:06.735356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:06.735317 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:06.863275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:06.863236 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:06.989447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:06.989358 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:07.067025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:07.066995 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4"] Oct 31 15:54:07.117924 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:07.117888 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c4840c-625h4"] Oct 31 15:54:07.305559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:07.305452 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:08.159803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:08.159734 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:08.289042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:08.289004 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:08.414385 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:08.414310 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:08.497743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:08.497713 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f545b0b7-1395-4c93-a084-9e8c2b58b759" path="/var/lib/kubelet/pods/f545b0b7-1395-4c93-a084-9e8c2b58b759/volumes" Oct 31 15:54:08.788598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:08.788567 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:08.794565 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:08.794538 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn"] Oct 31 15:54:08.794843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:08.794820 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" containerID="cri-o://96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f" gracePeriod=30 Oct 31 15:54:09.649233 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:09.649196 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:09.773599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:09.773566 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:09.897150 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:09.897113 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:10.210007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.209972 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj"] Oct 31 15:54:10.210798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.210776 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f36cb0c0-0bfb-4765-9959-44e4f514ee8d" containerName="revoke-keys" Oct 31 15:54:10.210798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.210798 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36cb0c0-0bfb-4765-9959-44e4f514ee8d" containerName="revoke-keys" Oct 31 15:54:10.211003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.210978 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36cb0c0-0bfb-4765-9959-44e4f514ee8d" containerName="revoke-keys" Oct 31 15:54:10.214771 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.214753 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.310129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.310096 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:10.327400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.326683 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9d627707-openshift-default-dockercfg-ptsrq\"" Oct 31 15:54:10.350495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.336248 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b670827b-dd4e-4e9f-938b-194b28436904-istio-podinfo\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.352641 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-socket\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.345969 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj"] Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.352752 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-certs\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.352809 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b670827b-dd4e-4e9f-938b-194b28436904-istiod-ca-cert\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.352936 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-istio-token\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.353054 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-credential-socket\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.353167 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-data\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.353424 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bwpp\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-kube-api-access-9bwpp\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.354649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.353512 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-envoy\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.454819 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.454779 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-socket\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.454836 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-certs\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.454886 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b670827b-dd4e-4e9f-938b-194b28436904-istiod-ca-cert\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.454960 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-istio-token\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455011 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-credential-socket\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455096 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-data\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455229 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bwpp\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-kube-api-access-9bwpp\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455272 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-envoy\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455304 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-certs\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455316 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b670827b-dd4e-4e9f-938b-194b28436904-istio-podinfo\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455224 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-socket\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455432 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-credential-socket\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455645 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455511 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-data\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.455761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.455644 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b670827b-dd4e-4e9f-938b-194b28436904-istiod-ca-cert\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.457702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.457672 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-envoy\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.457879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.457856 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b670827b-dd4e-4e9f-938b-194b28436904-istio-podinfo\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.514136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.514061 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-istio-token\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.514301 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.514162 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bwpp\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-kube-api-access-9bwpp\") pod \"e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.529169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.529145 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:10.582556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.582512 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:10.973274 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:54:10.973244 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb670827b_dd4e_4e9f_938b_194b28436904.slice/crio-c79c8b232f124ccf9b90e16c468f00f492dadcc852317e7499e8e1fcd0e0e172 WatchSource:0}: Error finding container c79c8b232f124ccf9b90e16c468f00f492dadcc852317e7499e8e1fcd0e0e172: Status 404 returned error can't find the container with id c79c8b232f124ccf9b90e16c468f00f492dadcc852317e7499e8e1fcd0e0e172 Oct 31 15:54:10.973822 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.973797 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj"] Oct 31 15:54:10.976068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.976035 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:54:10.976151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.976102 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:54:10.976151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:10.976130 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:54:11.778286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:11.778251 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" event={"ID":"b670827b-dd4e-4e9f-938b-194b28436904","Type":"ContainerStarted","Data":"de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9"} Oct 31 15:54:11.778286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:11.778310 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" event={"ID":"b670827b-dd4e-4e9f-938b-194b28436904","Type":"ContainerStarted","Data":"c79c8b232f124ccf9b90e16c468f00f492dadcc852317e7499e8e1fcd0e0e172"} Oct 31 15:54:11.900171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:11.900116 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" podStartSLOduration=1.9001017249999999 podStartE2EDuration="1.900101725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:54:11.895803166 +0000 UTC m=+4097.515354578" watchObservedRunningTime="2026-10-31 15:54:11.900101725 +0000 UTC m=+4097.519653199" Oct 31 15:54:11.982216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:11.982184 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:12.530448 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:12.530414 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:12.536663 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:12.536636 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:12.787039 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:12.786946 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:12.788108 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:12.788077 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:54:14.067008 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.066952 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:54:14.203824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.203795 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-envoy\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.203890 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22249730-1bc1-459b-b69a-5c83060c98c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22249730-1bc1-459b-b69a-5c83060c98c2-istio-podinfo\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.203911 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-socket\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.203972 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-credential-socket\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.203991 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-certs\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204037 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-data\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204127 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-kube-api-access-r8bgg\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-kube-api-access-r8bgg\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204202 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204161 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-istio-token\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-istio-token\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204221 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22249730-1bc1-459b-b69a-5c83060c98c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22249730-1bc1-459b-b69a-5c83060c98c2-istiod-ca-cert\") pod \"22249730-1bc1-459b-b69a-5c83060c98c2\" (UID: \"22249730-1bc1-459b-b69a-5c83060c98c2\") " Oct 31 15:54:14.204375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204277 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-credential-socket" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:14.204375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204294 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-socket" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:14.204375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204311 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-certs" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:14.204681 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204400 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-data" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:14.204891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204864 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.205021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204891 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22249730-1bc1-459b-b69a-5c83060c98c2-istiod-ca-cert" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:54:14.205021 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.204938 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.205151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.205076 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.205151 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.205091 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.206318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.206291 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22249730-1bc1-459b-b69a-5c83060c98c2-istio-podinfo" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:54:14.206421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.206325 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-envoy" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:14.206613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.206571 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-kube-api-access-r8bgg" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "kube-api-access-r8bgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:14.206613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.206607 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-istio-token" pod "22249730-1bc1-459b-b69a-5c83060c98c2" (UID: "22249730-1bc1-459b-b69a-5c83060c98c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:14.305997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.305960 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22249730-1bc1-459b-b69a-5c83060c98c2-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.305997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.305995 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22249730-1bc1-459b-b69a-5c83060c98c2-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.306271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.306011 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8bgg\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-kube-api-access-r8bgg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.306271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.306029 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22249730-1bc1-459b-b69a-5c83060c98c2-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.306271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.306043 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22249730-1bc1-459b-b69a-5c83060c98c2-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:14.628016 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.627986 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:14.762888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.762806 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:14.802254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.802220 4865 generic.go:356] "Generic (PLEG): container finished" podID="22249730-1bc1-459b-b69a-5c83060c98c2" containerID="96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f" exitCode=0 Oct 31 15:54:14.802437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.802291 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" Oct 31 15:54:14.802437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.802307 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" event={"ID":"22249730-1bc1-459b-b69a-5c83060c98c2","Type":"ContainerDied","Data":"96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f"} Oct 31 15:54:14.802437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.802393 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn" event={"ID":"22249730-1bc1-459b-b69a-5c83060c98c2","Type":"ContainerDied","Data":"675ceaa83b23d510375a8406a31778f0f6a40984ff9ba9b8681feb07e6894aee"} Oct 31 15:54:14.802437 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.802420 4865 scope.go:122] "RemoveContainer" containerID="96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f" Oct 31 15:54:14.813274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.813253 4865 scope.go:122] "RemoveContainer" containerID="0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5" Oct 31 15:54:14.822722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.822583 4865 scope.go:122] "RemoveContainer" containerID="96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f" Oct 31 15:54:14.822876 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:54:14.822856 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f\": container with ID starting with 96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f not found: ID does not exist" containerID="96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f" Oct 31 15:54:14.822923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.822884 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f"} err="failed to get container status \"96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f\": rpc error: code = NotFound desc = could not find container \"96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f\": container with ID starting with 96c8aa6cdc4a40175d5e2c71f3d8c9946baed4c38f357bcdf9502a5ef7ff550f not found: ID does not exist" Oct 31 15:54:14.822923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.822901 4865 scope.go:122] "RemoveContainer" containerID="0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5" Oct 31 15:54:14.823148 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:54:14.823129 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5\": container with ID starting with 0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5 not found: ID does not exist" containerID="0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5" Oct 31 15:54:14.823197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.823154 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5"} err="failed to get container status \"0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5\": rpc error: code = NotFound desc = could not find container \"0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5\": container with ID starting with 0c1b5dd3a1d34112417212e2577ed7adaaaf3f4534c01844fe3cdab1c91a09a5 not found: ID does not exist" Oct 31 15:54:14.888929 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.888905 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:14.972905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:14.972870 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn"] Oct 31 15:54:15.023885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:15.023808 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-73b8d8-openshift-default-d7849d79f-5txcn"] Oct 31 15:54:15.917206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:15.917172 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:16.497088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:16.497058 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" path="/var/lib/kubelet/pods/22249730-1bc1-459b-b69a-5c83060c98c2/volumes" Oct 31 15:54:16.647344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:16.647315 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s"] Oct 31 15:54:16.759446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:16.759367 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp"] Oct 31 15:54:16.998545 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:16.998518 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:17.122019 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:17.121951 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:17.237329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:17.237292 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:17.491012 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:54:17.490980 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:54:20.185394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.185358 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:54:20.387591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.387560 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:20.549036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.548978 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:20.807870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.807791 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5"] Oct 31 15:54:20.808451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.808437 4865 cpu_manager.go:414] "containerMap: removing container" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" Oct 31 15:54:20.808518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.808454 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" Oct 31 15:54:20.808617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.808607 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" Oct 31 15:54:20.808655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.808621 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" Oct 31 15:54:20.813674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.813656 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:20.943530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.922845 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:20.943530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.941687 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s944q\" (UniqueName: \"kubernetes.io/projected/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-kube-api-access-s944q\") pod \"payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:20.943530 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.943263 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5"] Oct 31 15:54:20.944212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.944186 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9d627707\"" Oct 31 15:54:20.944571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.944552 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9d627707-dockercfg-qqw4h\"" Oct 31 15:54:20.971210 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.970502 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx"] Oct 31 15:54:20.972598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.972568 4865 cpu_manager.go:414] "containerMap: removing container" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" Oct 31 15:54:20.972598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.972598 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22249730-1bc1-459b-b69a-5c83060c98c2" containerName="istio-proxy" Oct 31 15:54:20.977059 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:20.977038 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.012173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.012141 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx"] Oct 31 15:54:21.043781 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.043750 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqxj\" (UniqueName: \"kubernetes.io/projected/c6938d10-b7d4-4e0e-bc23-849e1ca85452-kube-api-access-6mqxj\") pod \"payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.043940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.043798 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:21.043989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.043930 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s944q\" (UniqueName: \"kubernetes.io/projected/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-kube-api-access-s944q\") pod \"payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:21.044033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.043989 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6938d10-b7d4-4e0e-bc23-849e1ca85452-plugins-config-volume\") pod \"payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.044590 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.044568 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:21.102369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.102301 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s944q\" (UniqueName: \"kubernetes.io/projected/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-kube-api-access-s944q\") pod \"payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:21.127831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.127805 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:21.145969 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.145941 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqxj\" (UniqueName: \"kubernetes.io/projected/c6938d10-b7d4-4e0e-bc23-849e1ca85452-kube-api-access-6mqxj\") pod \"payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.146111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.146085 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6938d10-b7d4-4e0e-bc23-849e1ca85452-plugins-config-volume\") pod \"payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.146731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.146711 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6938d10-b7d4-4e0e-bc23-849e1ca85452-plugins-config-volume\") pod \"payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.208818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.208790 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqxj\" (UniqueName: \"kubernetes.io/projected/c6938d10-b7d4-4e0e-bc23-849e1ca85452-kube-api-access-6mqxj\") pod \"payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.295857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.295823 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:21.511326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.511297 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5"] Oct 31 15:54:21.511452 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:54:21.511389 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ad7d6e_214b_45d7_ad3e_9e33b43af308.slice/crio-656516dd0a1b9988bac20ffb70f109760390a437ac183e743c8277c99d834328 WatchSource:0}: Error finding container 656516dd0a1b9988bac20ffb70f109760390a437ac183e743c8277c99d834328: Status 404 returned error can't find the container with id 656516dd0a1b9988bac20ffb70f109760390a437ac183e743c8277c99d834328 Oct 31 15:54:21.648115 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.648050 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" podUID="367cacad-404a-4a45-83e2-8ecf95bc9967" containerName="payload-processing" containerID="cri-o://7f3e4b2be2799be743c5d52c9719c58566ac480b18b405619f2c71532e707cb1" gracePeriod=25 Oct 31 15:54:21.686852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.686825 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:21.687428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.687406 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx"] Oct 31 15:54:21.709565 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:54:21.709534 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6938d10_b7d4_4e0e_bc23_849e1ca85452.slice/crio-8a5233784b96c8dd511d2104111a2f4d5d05475237bf3683147ec779b244cf73 WatchSource:0}: Error finding container 8a5233784b96c8dd511d2104111a2f4d5d05475237bf3683147ec779b244cf73: Status 404 returned error can't find the container with id 8a5233784b96c8dd511d2104111a2f4d5d05475237bf3683147ec779b244cf73 Oct 31 15:54:21.760589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.760554 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" podUID="5fa6c763-25d8-4f59-b390-8ead05aa3d7f" containerName="payload-pre-processing" containerID="cri-o://ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0" gracePeriod=25 Oct 31 15:54:21.855002 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.854964 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" event={"ID":"f3ad7d6e-214b-45d7-ad3e-9e33b43af308","Type":"ContainerStarted","Data":"1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606"} Oct 31 15:54:21.855178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.855024 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" event={"ID":"f3ad7d6e-214b-45d7-ad3e-9e33b43af308","Type":"ContainerStarted","Data":"656516dd0a1b9988bac20ffb70f109760390a437ac183e743c8277c99d834328"} Oct 31 15:54:21.855178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.855067 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:21.857129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.857103 4865 generic.go:356] "Generic (PLEG): container finished" podID="367cacad-404a-4a45-83e2-8ecf95bc9967" containerID="7f3e4b2be2799be743c5d52c9719c58566ac480b18b405619f2c71532e707cb1" exitCode=0 Oct 31 15:54:21.857258 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.857214 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" event={"ID":"367cacad-404a-4a45-83e2-8ecf95bc9967","Type":"ContainerDied","Data":"7f3e4b2be2799be743c5d52c9719c58566ac480b18b405619f2c71532e707cb1"} Oct 31 15:54:21.858790 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.858764 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" event={"ID":"c6938d10-b7d4-4e0e-bc23-849e1ca85452","Type":"ContainerStarted","Data":"8a5233784b96c8dd511d2104111a2f4d5d05475237bf3683147ec779b244cf73"} Oct 31 15:54:21.921030 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.920965 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:54:21.956096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.955342 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367cacad-404a-4a45-83e2-8ecf95bc9967-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/367cacad-404a-4a45-83e2-8ecf95bc9967-plugins-config-volume\") pod \"367cacad-404a-4a45-83e2-8ecf95bc9967\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " Oct 31 15:54:21.956096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.955631 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367cacad-404a-4a45-83e2-8ecf95bc9967-kube-api-access-47xpd\" (UniqueName: \"kubernetes.io/projected/367cacad-404a-4a45-83e2-8ecf95bc9967-kube-api-access-47xpd\") pod \"367cacad-404a-4a45-83e2-8ecf95bc9967\" (UID: \"367cacad-404a-4a45-83e2-8ecf95bc9967\") " Oct 31 15:54:21.956096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.955768 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367cacad-404a-4a45-83e2-8ecf95bc9967-plugins-config-volume" pod "367cacad-404a-4a45-83e2-8ecf95bc9967" (UID: "367cacad-404a-4a45-83e2-8ecf95bc9967"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:54:21.956362 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.956237 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/367cacad-404a-4a45-83e2-8ecf95bc9967-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:21.958435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:21.958409 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367cacad-404a-4a45-83e2-8ecf95bc9967-kube-api-access-47xpd" pod "367cacad-404a-4a45-83e2-8ecf95bc9967" (UID: "367cacad-404a-4a45-83e2-8ecf95bc9967"). InnerVolumeSpecName "kube-api-access-47xpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:22.031122 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.031098 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:54:22.057436 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.057414 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-kube-api-access-ghmxk\" (UniqueName: \"kubernetes.io/projected/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-kube-api-access-ghmxk\") pod \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " Oct 31 15:54:22.057644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.057629 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-plugins-config-volume\") pod \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\" (UID: \"5fa6c763-25d8-4f59-b390-8ead05aa3d7f\") " Oct 31 15:54:22.058037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.058016 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-plugins-config-volume" pod "5fa6c763-25d8-4f59-b390-8ead05aa3d7f" (UID: "5fa6c763-25d8-4f59-b390-8ead05aa3d7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:54:22.058157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.058143 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xpd\" (UniqueName: \"kubernetes.io/projected/367cacad-404a-4a45-83e2-8ecf95bc9967-kube-api-access-47xpd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:22.058203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.058161 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:22.059618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.059592 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-kube-api-access-ghmxk" pod "5fa6c763-25d8-4f59-b390-8ead05aa3d7f" (UID: "5fa6c763-25d8-4f59-b390-8ead05aa3d7f"). InnerVolumeSpecName "kube-api-access-ghmxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:22.096140 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.096017 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" podStartSLOduration=2.095957435 podStartE2EDuration="2.095957435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:54:21.972643718 +0000 UTC m=+4107.592195130" watchObservedRunningTime="2026-10-31 15:54:22.095957435 +0000 UTC m=+4107.715508849" Oct 31 15:54:22.159408 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.159341 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmxk\" (UniqueName: \"kubernetes.io/projected/5fa6c763-25d8-4f59-b390-8ead05aa3d7f-kube-api-access-ghmxk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:22.232074 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.232036 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:22.867057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.866967 4865 generic.go:356] "Generic (PLEG): container finished" podID="5fa6c763-25d8-4f59-b390-8ead05aa3d7f" containerID="ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0" exitCode=0 Oct 31 15:54:22.867057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.867044 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" Oct 31 15:54:22.867283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.867058 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" event={"ID":"5fa6c763-25d8-4f59-b390-8ead05aa3d7f","Type":"ContainerDied","Data":"ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0"} Oct 31 15:54:22.867283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.867143 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp" event={"ID":"5fa6c763-25d8-4f59-b390-8ead05aa3d7f","Type":"ContainerDied","Data":"46a2c22f4df1fe096cb48a8594389477d527e131da8a6d27555898ecedf8ddf0"} Oct 31 15:54:22.867283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.867172 4865 scope.go:122] "RemoveContainer" containerID="ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0" Oct 31 15:54:22.868848 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.868830 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" Oct 31 15:54:22.868968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.868839 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s" event={"ID":"367cacad-404a-4a45-83e2-8ecf95bc9967","Type":"ContainerDied","Data":"e5cdf2e37c26d111f9c9e35b5c51ba6833d16dee0257b54a201d0506a0fe4a48"} Oct 31 15:54:22.870886 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.870859 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" event={"ID":"c6938d10-b7d4-4e0e-bc23-849e1ca85452","Type":"ContainerStarted","Data":"61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390"} Oct 31 15:54:22.871024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.870990 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:22.880035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.880004 4865 scope.go:122] "RemoveContainer" containerID="ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0" Oct 31 15:54:22.880303 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:54:22.880287 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0\": container with ID starting with ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0 not found: ID does not exist" containerID="ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0" Oct 31 15:54:22.880358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.880323 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0"} err="failed to get container status \"ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0\": rpc error: code = NotFound desc = could not find container \"ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0\": container with ID starting with ef14473bda99683b032047005e61a51a78f5e347067cbdef1c52e1583be666d0 not found: ID does not exist" Oct 31 15:54:22.880358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:22.880340 4865 scope.go:122] "RemoveContainer" containerID="7f3e4b2be2799be743c5d52c9719c58566ac480b18b405619f2c71532e707cb1" Oct 31 15:54:23.035549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:23.035520 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s"] Oct 31 15:54:23.085981 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:23.085952 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c5ee32-6fdc99c789-8rg9s"] Oct 31 15:54:23.196401 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:23.196314 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp"] Oct 31 15:54:23.199699 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:23.199673 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c5ee32-6d489c495-dwmwp"] Oct 31 15:54:23.312946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:23.312895 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" podStartSLOduration=3.312878613 podStartE2EDuration="3.312878613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:54:23.310797013 +0000 UTC m=+4108.930348426" watchObservedRunningTime="2026-10-31 15:54:23.312878613 +0000 UTC m=+4108.932430032" Oct 31 15:54:24.498332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:24.498301 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367cacad-404a-4a45-83e2-8ecf95bc9967" path="/var/lib/kubelet/pods/367cacad-404a-4a45-83e2-8ecf95bc9967/volumes" Oct 31 15:54:24.498800 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:24.498783 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa6c763-25d8-4f59-b390-8ead05aa3d7f" path="/var/lib/kubelet/pods/5fa6c763-25d8-4f59-b390-8ead05aa3d7f/volumes" Oct 31 15:54:26.995287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:26.995250 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:30.561317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:30.561282 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:31.634896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:31.634861 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:32.874808 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:32.874771 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:54:33.884656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:33.884627 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:54:36.304941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:36.304907 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:36.514003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:36.513971 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:40.910293 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:40.910262 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:44.972548 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:44.972512 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:46.621155 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:46.621097 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:46.752186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:46.752158 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:46.845004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:46.844974 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58"] Oct 31 15:54:46.895099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:46.895028 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-883697-ghn58"] Oct 31 15:54:47.138254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:47.138211 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:47.491009 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:54:47.490972 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:54:47.829536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:47.829504 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:48.497850 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:48.497820 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36cb0c0-0bfb-4765-9959-44e4f514ee8d" path="/var/lib/kubelet/pods/f36cb0c0-0bfb-4765-9959-44e4f514ee8d/volumes" Oct 31 15:54:49.024347 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:49.024309 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:49.158960 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:49.158928 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:50.369953 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:50.369921 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:51.627619 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:51.627582 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:53.507305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:53.507271 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:54.557799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:54.557764 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq"] Oct 31 15:54:54.899510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:54.899414 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:55.049624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:55.049585 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:55.194186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:55.194109 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:55.867994 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:55.867958 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:56.768936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:56.768901 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:57.808443 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:57.808407 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:59.281444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.281402 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:59.558203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.558107 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" podUID="05e2621d-8790-49d7-8394-f4aaca957c67" containerName="payload-processing" containerID="cri-o://934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69" gracePeriod=25 Oct 31 15:54:59.827095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.827021 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:54:59.853068 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.853033 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e2621d-8790-49d7-8394-f4aaca957c67-kube-api-access-fqzgl\" (UniqueName: \"kubernetes.io/projected/05e2621d-8790-49d7-8394-f4aaca957c67-kube-api-access-fqzgl\") pod \"05e2621d-8790-49d7-8394-f4aaca957c67\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " Oct 31 15:54:59.853284 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.853270 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e2621d-8790-49d7-8394-f4aaca957c67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e2621d-8790-49d7-8394-f4aaca957c67-plugins-config-volume\") pod \"05e2621d-8790-49d7-8394-f4aaca957c67\" (UID: \"05e2621d-8790-49d7-8394-f4aaca957c67\") " Oct 31 15:54:59.853656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.853630 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e2621d-8790-49d7-8394-f4aaca957c67-plugins-config-volume" pod "05e2621d-8790-49d7-8394-f4aaca957c67" (UID: "05e2621d-8790-49d7-8394-f4aaca957c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:54:59.855383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.855361 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e2621d-8790-49d7-8394-f4aaca957c67-kube-api-access-fqzgl" pod "05e2621d-8790-49d7-8394-f4aaca957c67" (UID: "05e2621d-8790-49d7-8394-f4aaca957c67"). InnerVolumeSpecName "kube-api-access-fqzgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:59.868966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.868933 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:54:59.954393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.954364 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqzgl\" (UniqueName: \"kubernetes.io/projected/05e2621d-8790-49d7-8394-f4aaca957c67-kube-api-access-fqzgl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:59.954393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:54:59.954391 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e2621d-8790-49d7-8394-f4aaca957c67-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:00.138381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.138293 4865 generic.go:356] "Generic (PLEG): container finished" podID="05e2621d-8790-49d7-8394-f4aaca957c67" containerID="934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69" exitCode=0 Oct 31 15:55:00.138381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.138342 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" Oct 31 15:55:00.138675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.138388 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" event={"ID":"05e2621d-8790-49d7-8394-f4aaca957c67","Type":"ContainerDied","Data":"934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69"} Oct 31 15:55:00.138675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.138447 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq" event={"ID":"05e2621d-8790-49d7-8394-f4aaca957c67","Type":"ContainerDied","Data":"4a0675264acd7416fe4dfc686c3d04222b2046155ee95f0d43df8b71d2649808"} Oct 31 15:55:00.138675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.138511 4865 scope.go:122] "RemoveContainer" containerID="934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69" Oct 31 15:55:00.150114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.150093 4865 scope.go:122] "RemoveContainer" containerID="934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69" Oct 31 15:55:00.150444 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:55:00.150423 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69\": container with ID starting with 934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69 not found: ID does not exist" containerID="934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69" Oct 31 15:55:00.150556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.150455 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69"} err="failed to get container status \"934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69\": rpc error: code = NotFound desc = could not find container \"934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69\": container with ID starting with 934fa70e51ca86524328becb2a7ec97545faee92b00ab2defbd5ddc8ea427b69 not found: ID does not exist" Oct 31 15:55:00.308197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.308168 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq"] Oct 31 15:55:00.311505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.311463 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf95c9-655f68bdf-jmhpq"] Oct 31 15:55:00.501015 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:00.500980 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e2621d-8790-49d7-8394-f4aaca957c67" path="/var/lib/kubelet/pods/05e2621d-8790-49d7-8394-f4aaca957c67/volumes" Oct 31 15:55:01.582674 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:01.582630 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:01.702918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:01.702890 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:02.513741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:02.513708 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:05.896046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:05.896012 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:07.219004 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:07.218955 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:08.425238 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:08.425200 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:55:08.665165 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:08.665117 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:10.195898 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:10.195856 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:11.887570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:11.887532 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:12.031519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:12.031444 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:16.515435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:16.515400 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:17.491104 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:55:17.491070 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:55:20.672823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:20.672785 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:22.412540 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:22.412463 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:27.044092 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:27.044014 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:27.204524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:27.204430 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:27.334462 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:27.334383 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:27.498148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:27.498119 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:27.739178 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:27.739146 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:29.033594 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:29.033554 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:29.695491 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:29.695445 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:32.214868 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:32.214836 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:36.883797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:36.883708 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:39.665964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:39.665930 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:55:41.543610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:41.543571 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:46.230524 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:46.230483 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:47.493560 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:55:47.493522 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:55:50.558816 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:50.558778 4865 generic.go:356] "Generic (PLEG): container finished" podID="b670827b-dd4e-4e9f-938b-194b28436904" containerID="de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9" exitCode=0 Oct 31 15:55:50.559260 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:50.558882 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" event={"ID":"b670827b-dd4e-4e9f-938b-194b28436904","Type":"ContainerDied","Data":"de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9"} Oct 31 15:55:50.559309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:50.559269 4865 scope.go:122] "RemoveContainer" containerID="de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9" Oct 31 15:55:50.560897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:50.560865 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:55:50.561018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:50.560954 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:55:50.561018 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:50.561001 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:55:50.916618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:50.916524 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:51.569188 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:51.569155 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" event={"ID":"b670827b-dd4e-4e9f-938b-194b28436904","Type":"ContainerStarted","Data":"48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743"} Oct 31 15:55:52.529534 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:52.529500 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:55:52.529752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:52.529653 4865 patch_prober.go:29] interesting pod/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 31 15:55:52.529752 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:52.529730 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 31 15:55:53.529738 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:53.529699 4865 patch_prober.go:29] interesting pod/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 31 15:55:53.530246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:53.529770 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 31 15:55:54.529446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:54.529413 4865 patch_prober.go:29] interesting pod/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 31 15:55:54.529656 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:54.529463 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 31 15:55:55.530246 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:55.530206 4865 patch_prober.go:29] interesting pod/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 31 15:55:55.530631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:55.530266 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 31 15:55:55.571532 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:55.571502 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:55:56.174617 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:56.174592 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:55:56.183118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:56.183093 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:55:56.529718 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:56.529686 4865 patch_prober.go:29] interesting pod/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Oct 31 15:55:56.529880 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:56.529735 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Oct 31 15:55:57.535073 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:57.535037 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:55:57.535608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:57.535411 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:55:57.536247 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:57.536224 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:55:58.619897 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:58.619872 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 15:55:58.627423 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:55:58.627404 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 15:56:00.340761 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:00.340726 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:01.053124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:01.053084 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:01.196766 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:01.196732 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:01.339174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:01.339088 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:02.444853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:02.444817 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:02.578225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:02.578192 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:02.719650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:02.719615 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:04.554430 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.554402 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8"] Oct 31 15:56:04.555093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555069 4865 cpu_manager.go:414] "containerMap: removing container" podUID="05e2621d-8790-49d7-8394-f4aaca957c67" containerName="payload-processing" Oct 31 15:56:04.555093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555086 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e2621d-8790-49d7-8394-f4aaca957c67" containerName="payload-processing" Oct 31 15:56:04.555200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555110 4865 cpu_manager.go:414] "containerMap: removing container" podUID="367cacad-404a-4a45-83e2-8ecf95bc9967" containerName="payload-processing" Oct 31 15:56:04.555200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555117 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367cacad-404a-4a45-83e2-8ecf95bc9967" containerName="payload-processing" Oct 31 15:56:04.555200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555126 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5fa6c763-25d8-4f59-b390-8ead05aa3d7f" containerName="payload-pre-processing" Oct 31 15:56:04.555200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555131 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa6c763-25d8-4f59-b390-8ead05aa3d7f" containerName="payload-pre-processing" Oct 31 15:56:04.555335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555246 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e2621d-8790-49d7-8394-f4aaca957c67" containerName="payload-processing" Oct 31 15:56:04.555335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555259 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="367cacad-404a-4a45-83e2-8ecf95bc9967" containerName="payload-processing" Oct 31 15:56:04.555335 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.555272 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa6c763-25d8-4f59-b390-8ead05aa3d7f" containerName="payload-pre-processing" Oct 31 15:56:04.558657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.558639 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.613823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.613798 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8"] Oct 31 15:56:04.654225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.654196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.654394 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.654318 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhzg\" (UniqueName: \"kubernetes.io/projected/acfc645f-fea4-4c6a-9e04-f4f384183abb-kube-api-access-4hhzg\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.654446 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.654432 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.660765 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.660739 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:56:04.660895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.660782 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:56:04.660895 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.660826 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:56:04.662559 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.662545 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:56:04.755913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.755887 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.756081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.755947 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhzg\" (UniqueName: \"kubernetes.io/projected/acfc645f-fea4-4c6a-9e04-f4f384183abb-kube-api-access-4hhzg\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.756081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.756042 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.756577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.756555 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.758528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.758511 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.813842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.813782 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhzg\" (UniqueName: \"kubernetes.io/projected/acfc645f-fea4-4c6a-9e04-f4f384183abb-kube-api-access-4hhzg\") pod \"maas-api-revoke-sub-82a3df2e-kz4r8\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.872251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.872226 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:04.981453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:04.981416 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:05.248124 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:05.248101 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8"] Oct 31 15:56:05.248516 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:56:05.248456 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfc645f_fea4_4c6a_9e04_f4f384183abb.slice/crio-cf39bf9d9edfd8512c5a905bc261dbc5554f609ad9ff5196999634b444b20c5f WatchSource:0}: Error finding container cf39bf9d9edfd8512c5a905bc261dbc5554f609ad9ff5196999634b444b20c5f: Status 404 returned error can't find the container with id cf39bf9d9edfd8512c5a905bc261dbc5554f609ad9ff5196999634b444b20c5f Oct 31 15:56:05.695158 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:05.695119 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" event={"ID":"acfc645f-fea4-4c6a-9e04-f4f384183abb","Type":"ContainerStarted","Data":"893642d7c951dab3dbd4fe2f722a4a2c0a2a6e1575ec6fea9fca236d6f695f01"} Oct 31 15:56:05.695696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:05.695215 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" event={"ID":"acfc645f-fea4-4c6a-9e04-f4f384183abb","Type":"ContainerStarted","Data":"cf39bf9d9edfd8512c5a905bc261dbc5554f609ad9ff5196999634b444b20c5f"} Oct 31 15:56:05.812767 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:05.812655 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" podStartSLOduration=1.812640848 podStartE2EDuration="1.812640848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:56:05.811056986 +0000 UTC m=+4211.430608398" watchObservedRunningTime="2026-10-31 15:56:05.812640848 +0000 UTC m=+4211.432192259" Oct 31 15:56:06.704364 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:06.704331 4865 generic.go:356] "Generic (PLEG): container finished" podID="acfc645f-fea4-4c6a-9e04-f4f384183abb" containerID="893642d7c951dab3dbd4fe2f722a4a2c0a2a6e1575ec6fea9fca236d6f695f01" exitCode=0 Oct 31 15:56:06.704810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:06.704413 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" event={"ID":"acfc645f-fea4-4c6a-9e04-f4f384183abb","Type":"ContainerDied","Data":"893642d7c951dab3dbd4fe2f722a4a2c0a2a6e1575ec6fea9fca236d6f695f01"} Oct 31 15:56:07.864873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.864848 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:07.892388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.892358 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-xks-ca\") pod \"acfc645f-fea4-4c6a-9e04-f4f384183abb\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " Oct 31 15:56:07.892569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.892507 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-openshift-ca\") pod \"acfc645f-fea4-4c6a-9e04-f4f384183abb\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " Oct 31 15:56:07.892673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.892660 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfc645f-fea4-4c6a-9e04-f4f384183abb-kube-api-access-4hhzg\" (UniqueName: \"kubernetes.io/projected/acfc645f-fea4-4c6a-9e04-f4f384183abb-kube-api-access-4hhzg\") pod \"acfc645f-fea4-4c6a-9e04-f4f384183abb\" (UID: \"acfc645f-fea4-4c6a-9e04-f4f384183abb\") " Oct 31 15:56:07.892907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.892879 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-openshift-ca" pod "acfc645f-fea4-4c6a-9e04-f4f384183abb" (UID: "acfc645f-fea4-4c6a-9e04-f4f384183abb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:56:07.893198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.893090 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:56:07.894636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.894613 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-xks-ca" pod "acfc645f-fea4-4c6a-9e04-f4f384183abb" (UID: "acfc645f-fea4-4c6a-9e04-f4f384183abb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:56:07.894788 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.894773 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfc645f-fea4-4c6a-9e04-f4f384183abb-kube-api-access-4hhzg" pod "acfc645f-fea4-4c6a-9e04-f4f384183abb" (UID: "acfc645f-fea4-4c6a-9e04-f4f384183abb"). InnerVolumeSpecName "kube-api-access-4hhzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:56:07.994383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.994305 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhzg\" (UniqueName: \"kubernetes.io/projected/acfc645f-fea4-4c6a-9e04-f4f384183abb-kube-api-access-4hhzg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:56:07.994383 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:07.994336 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acfc645f-fea4-4c6a-9e04-f4f384183abb-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:56:08.720729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:08.720696 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" Oct 31 15:56:08.720729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:08.720717 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8" event={"ID":"acfc645f-fea4-4c6a-9e04-f4f384183abb","Type":"ContainerDied","Data":"cf39bf9d9edfd8512c5a905bc261dbc5554f609ad9ff5196999634b444b20c5f"} Oct 31 15:56:08.720939 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:08.720772 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf39bf9d9edfd8512c5a905bc261dbc5554f609ad9ff5196999634b444b20c5f" Oct 31 15:56:09.036086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:09.036049 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:10.503357 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:10.503326 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:10.685439 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:10.685405 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:13.158084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:13.158053 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:13.293914 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:13.293882 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:14.611614 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:14.611578 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:15.075229 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:15.075196 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:15.296865 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:15.296830 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:15.490356 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:15.490320 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:15.821851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:15.821768 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:16.313407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.313377 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf"] Oct 31 15:56:16.314114 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.314096 4865 cpu_manager.go:414] "containerMap: removing container" podUID="acfc645f-fea4-4c6a-9e04-f4f384183abb" containerName="revoke-keys" Oct 31 15:56:16.314186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.314115 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfc645f-fea4-4c6a-9e04-f4f384183abb" containerName="revoke-keys" Oct 31 15:56:16.314269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.314258 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfc645f-fea4-4c6a-9e04-f4f384183abb" containerName="revoke-keys" Oct 31 15:56:16.318109 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.318089 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.393731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.393690 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf"] Oct 31 15:56:16.395253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.395224 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:16.396707 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.396682 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-podinfo\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.396836 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.396750 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-certs\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.396931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.396904 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istiod-ca-cert\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.397025 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.397006 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-data\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.397064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.397048 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hfd\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-kube-api-access-g5hfd\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.397126 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.397115 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.397186 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.397169 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-token\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.397224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.397204 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-envoy\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.397257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.397238 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-credential-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.423602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.423576 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-31dff8-openshift-default-dockercfg-5k2pv\"" Oct 31 15:56:16.514118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.514080 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-podinfo\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.514506 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.514454 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-certs\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.514749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.514729 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istiod-ca-cert\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.515035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.514903 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-data\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.515223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.515193 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hfd\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-kube-api-access-g5hfd\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.515432 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.515414 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.515618 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.515598 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-token\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.515862 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.515835 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-envoy\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.516049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.516030 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-credential-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.516676 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.516652 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-credential-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.518425 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.518403 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-data\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.529042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.519600 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istiod-ca-cert\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.529042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.522061 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-certs\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.529042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.522166 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.529042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.527644 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-podinfo\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.529042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.528880 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-envoy\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.581938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.581857 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hfd\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-kube-api-access-g5hfd\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.581938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.581927 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-token\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.632943 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.632911 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:16.747978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:16.747943 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:17.068052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:17.068025 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf"] Oct 31 15:56:17.069481 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:56:17.069436 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de2ffbe_06bd_4a52_8bf2_b583cd1a8a21.slice/crio-821bae8a431d3f735bbe70577a820b8f37624d4a4639093377d46aeec7302eb8 WatchSource:0}: Error finding container 821bae8a431d3f735bbe70577a820b8f37624d4a4639093377d46aeec7302eb8: Status 404 returned error can't find the container with id 821bae8a431d3f735bbe70577a820b8f37624d4a4639093377d46aeec7302eb8 Oct 31 15:56:17.072031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:17.071997 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:56:17.072138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:17.072064 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:56:17.072138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:17.072104 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:56:17.491714 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:56:17.491675 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:56:17.790306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:17.790208 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" event={"ID":"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21","Type":"ContainerStarted","Data":"cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97"} Oct 31 15:56:17.790306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:17.790269 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" event={"ID":"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21","Type":"ContainerStarted","Data":"821bae8a431d3f735bbe70577a820b8f37624d4a4639093377d46aeec7302eb8"} Oct 31 15:56:17.960576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:17.960523 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" podStartSLOduration=1.960509435 podStartE2EDuration="1.960509435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:56:17.958505821 +0000 UTC m=+4223.578057232" watchObservedRunningTime="2026-10-31 15:56:17.960509435 +0000 UTC m=+4223.580060846" Oct 31 15:56:18.633538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:18.633503 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:18.639803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:18.639776 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:18.798033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:18.797998 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:18.798892 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:18.798870 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:56:20.630500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:20.630333 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:20.757303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:20.756014 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:20.912217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:20.912126 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:22.446495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:22.446361 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:22.684495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:22.684248 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:23.294495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:23.293792 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:56:23.294495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:23.293890 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:23.316495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:23.316330 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:56:23.316495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:23.316442 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:23.635550 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:23.634577 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:23.972794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:23.972669 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:56:23.972794 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:23.972755 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:24.014486 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:24.014439 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:24.142084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:24.142047 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:25.698003 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:25.697965 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:30.402949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:30.402909 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:35.185748 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:35.185688 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:36.359453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:36.359395 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:36.509820 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:36.509744 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:36.802391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:36.802358 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:39.926071 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:39.926026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:42.386081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:42.386024 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:42.519934 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:42.519902 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:42.814658 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:42.814626 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:44.589290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:44.589258 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:47.587459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:47.587348 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:47.692774 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:56:47.692710 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:56:48.375157 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:48.375126 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:48.564637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:48.564596 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:48.599498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:48.599092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:48.697591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:48.697500 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:48.964064 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:48.964030 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:49.165851 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:49.165798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:49.371557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:49.371448 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:56:49.380823 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:49.377457 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:56:49.419720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:49.419681 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:49.543117 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:49.543061 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:49.979344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:49.979288 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:54.770949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:54.770913 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:55.092243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:55.092168 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:55.240096 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:55.240052 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:56:59.881739 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:56:59.881690 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:04.503935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:04.503892 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:08.547269 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:08.547150 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:09.402492 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:09.397430 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:10.297636 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:10.297598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:10.860318 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:10.860274 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:14.995101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:14.995069 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:17.490157 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:17.490120 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:57:17.857633 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:17.857558 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:18.065649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:18.065617 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:19.417296 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:19.417256 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:19.839374 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:19.839343 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:19.884372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:19.884339 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw"] Oct 31 15:57:19.884599 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:19.884579 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" containerID="cri-o://4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a" gracePeriod=30 Oct 31 15:57:20.000111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:19.999675 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:20.575536 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:20.575508 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:24.581525 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:24.581458 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:25.162005 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.161939 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:57:25.265555 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265527 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-data\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.265719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265610 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-certs\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.265719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265674 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istiod-ca-cert\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.265719 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265711 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-podinfo\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.265894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265732 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-credential-socket\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.265894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265789 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-data" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:25.265894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265864 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-token\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-token\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.265894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265866 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-certs" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:25.266136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.265944 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-envoy\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.266136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266011 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-credential-socket" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:25.266136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266057 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-socket\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.266136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266099 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-kube-api-access-crc4m\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-kube-api-access-crc4m\") pod \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\" (UID: \"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851\") " Oct 31 15:57:25.266136 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266106 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istiod-ca-cert" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:25.266441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266349 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-socket" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:25.266785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266762 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.266785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266788 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.267138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266803 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.267138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266819 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.267138 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.266833 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.268225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.268204 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-podinfo" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:57:25.268521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.268496 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-token" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:25.268621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.268600 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-envoy" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:25.268671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.268638 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-kube-api-access-crc4m" pod "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" (UID: "3e9b2ee2-3aa1-4f79-98d8-cf7690b43851"). InnerVolumeSpecName "kube-api-access-crc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:25.368346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.368317 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.368346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.368343 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.368668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.368353 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crc4m\" (UniqueName: \"kubernetes.io/projected/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-kube-api-access-crc4m\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.368668 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.368363 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:25.553179 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.553141 4865 generic.go:356] "Generic (PLEG): container finished" podID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerID="4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a" exitCode=0 Oct 31 15:57:25.553336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.553198 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" Oct 31 15:57:25.553336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.553221 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" event={"ID":"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851","Type":"ContainerDied","Data":"4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a"} Oct 31 15:57:25.553336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.553274 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw" event={"ID":"3e9b2ee2-3aa1-4f79-98d8-cf7690b43851","Type":"ContainerDied","Data":"a5b30f8b8f1e7861b05d78017ea9a89c3417c09359385b97226d77893b15a65f"} Oct 31 15:57:25.553336 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.553295 4865 scope.go:122] "RemoveContainer" containerID="4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a" Oct 31 15:57:25.567806 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.565219 4865 scope.go:122] "RemoveContainer" containerID="d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03" Oct 31 15:57:25.577428 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.577408 4865 scope.go:122] "RemoveContainer" containerID="4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a" Oct 31 15:57:25.577731 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:25.577711 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a\": container with ID starting with 4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a not found: ID does not exist" containerID="4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a" Oct 31 15:57:25.577821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.577739 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a"} err="failed to get container status \"4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a\": rpc error: code = NotFound desc = could not find container \"4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a\": container with ID starting with 4f11484e61d5584210281387017555d92fc63ea28b3b4f89e13133964dfa205a not found: ID does not exist" Oct 31 15:57:25.577821 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.577757 4865 scope.go:122] "RemoveContainer" containerID="d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03" Oct 31 15:57:25.578017 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:25.577999 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03\": container with ID starting with d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03 not found: ID does not exist" containerID="d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03" Oct 31 15:57:25.578065 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.578026 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03"} err="failed to get container status \"d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03\": rpc error: code = NotFound desc = could not find container \"d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03\": container with ID starting with d701dd7d6f94da97e68ac499acab7a11275be8386e4102cecc056dfcfd916e03 not found: ID does not exist" Oct 31 15:57:25.724326 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.724292 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw"] Oct 31 15:57:25.728213 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:25.728193 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-21111f-openshift-default-f9d4b75c5-pcxmw"] Oct 31 15:57:26.050912 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:26.050883 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd"] Oct 31 15:57:26.053787 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:26.053767 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ef4ab-52xhd"] Oct 31 15:57:26.496919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:26.496886 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f82d107-968b-4d89-9f9f-55051c595f80" path="/var/lib/kubelet/pods/2f82d107-968b-4d89-9f9f-55051c595f80/volumes" Oct 31 15:57:26.497366 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:26.497349 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" path="/var/lib/kubelet/pods/3e9b2ee2-3aa1-4f79-98d8-cf7690b43851/volumes" Oct 31 15:57:26.685937 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:26.685907 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:27.036048 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:27.036016 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl"] Oct 31 15:57:27.184232 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:27.184192 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7"] Oct 31 15:57:27.396190 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:27.396104 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:27.515095 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:27.515063 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:29.462916 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:29.462853 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:29.630494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:29.630435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:29.754452 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:29.754371 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:30.962348 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:30.962314 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:57:32.037224 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:32.037161 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" podUID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" containerName="payload-processing" containerID="cri-o://70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612" gracePeriod=25 Oct 31 15:57:32.185407 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:32.185367 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" podUID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" containerName="payload-pre-processing" containerID="cri-o://db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0" gracePeriod=25 Oct 31 15:57:32.305533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:32.305437 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:34.013904 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:34.013874 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:35.820584 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:35.820548 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:37.972962 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:37.972926 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:38.106747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:38.106708 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:39.508372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:39.508342 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:39.962368 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:39.962331 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:39.972459 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:39.972431 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9"] Oct 31 15:57:39.972712 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:39.972687 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" containerID="cri-o://5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980" gracePeriod=30 Oct 31 15:57:40.395203 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.395111 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:40.645342 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.645265 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:40.976038 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.976006 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w"] Oct 31 15:57:40.976716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.976699 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" Oct 31 15:57:40.976716 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.976717 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" Oct 31 15:57:40.976857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.976734 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" Oct 31 15:57:40.976857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.976739 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" Oct 31 15:57:40.976857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.976826 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" Oct 31 15:57:40.976857 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.976842 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9b2ee2-3aa1-4f79-98d8-cf7690b43851" containerName="istio-proxy" Oct 31 15:57:40.980557 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:40.980540 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.035125 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.035089 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w"] Oct 31 15:57:41.041166 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.041138 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:57:41.041328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.041190 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:57:41.041328 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.041229 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:57:41.081042 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.081019 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgb6\" (UniqueName: \"kubernetes.io/projected/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-kube-api-access-mhgb6\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.081177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.081071 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.081177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.081091 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.081649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.081630 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:57:41.182036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.182007 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgb6\" (UniqueName: \"kubernetes.io/projected/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-kube-api-access-mhgb6\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.182174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.182061 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.182174 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.182080 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.182802 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.182775 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.184721 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.184695 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.239069 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.239014 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgb6\" (UniqueName: \"kubernetes.io/projected/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-kube-api-access-mhgb6\") pod \"maas-api-revoke-sub-369e434a-kzr8w\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.295040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.295012 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:41.562833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.562753 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" podUID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:57:41.562833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.562804 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" podUID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:57:41.672045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:41.672015 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w"] Oct 31 15:57:41.672628 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:57:41.672598 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57b8bb7_dd26_49c7_b98d_d8e4afec0b28.slice/crio-131d6b441d1f2a875f3247888b4e0614eac00600fc17efa0bf1709fd17819947 WatchSource:0}: Error finding container 131d6b441d1f2a875f3247888b4e0614eac00600fc17efa0bf1709fd17819947: Status 404 returned error can't find the container with id 131d6b441d1f2a875f3247888b4e0614eac00600fc17efa0bf1709fd17819947 Oct 31 15:57:42.689818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:42.689787 4865 generic.go:356] "Generic (PLEG): container finished" podID="f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" containerID="64630c79c771d0b65950ca8b363e80134a4cd6ccf90a9d18c10bcb8c9c5fccb7" exitCode=0 Oct 31 15:57:42.690205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:42.689841 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" event={"ID":"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28","Type":"ContainerDied","Data":"64630c79c771d0b65950ca8b363e80134a4cd6ccf90a9d18c10bcb8c9c5fccb7"} Oct 31 15:57:42.690205 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:42.689890 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" event={"ID":"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28","Type":"ContainerStarted","Data":"131d6b441d1f2a875f3247888b4e0614eac00600fc17efa0bf1709fd17819947"} Oct 31 15:57:43.860695 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:43.860670 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:44.018689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.018656 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-openshift-ca\") pod \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " Oct 31 15:57:44.018888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.018778 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-kube-api-access-mhgb6\" (UniqueName: \"kubernetes.io/projected/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-kube-api-access-mhgb6\") pod \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " Oct 31 15:57:44.018888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.018798 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-xks-ca\") pod \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\" (UID: \"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28\") " Oct 31 15:57:44.019083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.019057 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-openshift-ca" pod "f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" (UID: "f57b8bb7-dd26-49c7-b98d-d8e4afec0b28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:44.019309 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.019292 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:44.021156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.021134 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-kube-api-access-mhgb6" pod "f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" (UID: "f57b8bb7-dd26-49c7-b98d-d8e4afec0b28"). InnerVolumeSpecName "kube-api-access-mhgb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:44.021156 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.021142 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-xks-ca" pod "f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" (UID: "f57b8bb7-dd26-49c7-b98d-d8e4afec0b28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:57:44.120877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.120848 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhgb6\" (UniqueName: \"kubernetes.io/projected/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-kube-api-access-mhgb6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:44.120877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.120875 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:44.705833 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.705805 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" Oct 31 15:57:44.706011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.705805 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w" event={"ID":"f57b8bb7-dd26-49c7-b98d-d8e4afec0b28","Type":"ContainerDied","Data":"131d6b441d1f2a875f3247888b4e0614eac00600fc17efa0bf1709fd17819947"} Oct 31 15:57:44.706011 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:44.705924 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="131d6b441d1f2a875f3247888b4e0614eac00600fc17efa0bf1709fd17819947" Oct 31 15:57:45.243901 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.243849 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:57:45.334384 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334354 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-socket\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334422 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-certs\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334450 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-kube-api-access-bglpj\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-kube-api-access-bglpj\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334527 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-envoy\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334571 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334563 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-podinfo\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334626 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-data\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334677 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istiod-ca-cert\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334690 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-socket" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:45.334872 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334733 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-certs" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:45.334925 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334886 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-credential-socket\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.334978 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334932 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-token\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-token\") pod \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\" (UID: \"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d\") " Oct 31 15:57:45.335099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.334933 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-data" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:45.335099 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.335085 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-credential-socket" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:45.335290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.335042 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istiod-ca-cert" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:45.335689 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.335623 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.335824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.335781 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.335899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.335828 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.335899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.335842 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.335899 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.335858 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.337054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.337029 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-podinfo" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:57:45.337185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.337106 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-token" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:45.337185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.337120 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-envoy" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:57:45.337185 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.337136 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-kube-api-access-bglpj" pod "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" (UID: "ee4a93c2-2dc1-4b9a-8cf0-821630eea00d"). InnerVolumeSpecName "kube-api-access-bglpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:45.361303 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.361276 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw"] Oct 31 15:57:45.437620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.437589 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bglpj\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-kube-api-access-bglpj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.437620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.437617 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.437620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.437626 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.437854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.437634 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:45.446149 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.446123 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:45.716033 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.715998 4865 generic.go:356] "Generic (PLEG): container finished" podID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerID="5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980" exitCode=0 Oct 31 15:57:45.716209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.716049 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" Oct 31 15:57:45.716209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.716084 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" event={"ID":"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d","Type":"ContainerDied","Data":"5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980"} Oct 31 15:57:45.716209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.716142 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9" event={"ID":"ee4a93c2-2dc1-4b9a-8cf0-821630eea00d","Type":"ContainerDied","Data":"d44514fdc013e463a267d92749f0fea4416b3c5b9a3b2dbd705e0544387e344f"} Oct 31 15:57:45.716209 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.716171 4865 scope.go:122] "RemoveContainer" containerID="5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980" Oct 31 15:57:45.720163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.720126 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:45.729955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.729933 4865 scope.go:122] "RemoveContainer" containerID="f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593" Oct 31 15:57:45.739456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.739437 4865 scope.go:122] "RemoveContainer" containerID="5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980" Oct 31 15:57:45.739786 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:45.739765 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980\": container with ID starting with 5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980 not found: ID does not exist" containerID="5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980" Oct 31 15:57:45.739854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.739793 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980"} err="failed to get container status \"5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980\": rpc error: code = NotFound desc = could not find container \"5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980\": container with ID starting with 5ecd77748143f625a41f3c1337ccc7fca5452ba09ae579d6bf9aeaa48e2ac980 not found: ID does not exist" Oct 31 15:57:45.739854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.739811 4865 scope.go:122] "RemoveContainer" containerID="f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593" Oct 31 15:57:45.740054 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:45.740037 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593\": container with ID starting with f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593 not found: ID does not exist" containerID="f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593" Oct 31 15:57:45.740093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.740061 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593"} err="failed to get container status \"f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593\": rpc error: code = NotFound desc = could not find container \"f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593\": container with ID starting with f5f9bc27114a880deff0ce07cbb6e85b4f33e1940ea2d6aac5e4f68682716593 not found: ID does not exist" Oct 31 15:57:45.893274 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.893238 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9"] Oct 31 15:57:45.895700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:45.895671 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-46bf36-openshift-default-75bf85f687-hnqb9"] Oct 31 15:57:46.251809 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.251774 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj"] Oct 31 15:57:46.252502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252455 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" containerName="revoke-keys" Oct 31 15:57:46.252502 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252485 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" containerName="revoke-keys" Oct 31 15:57:46.252598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252507 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" Oct 31 15:57:46.252598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252512 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" Oct 31 15:57:46.252598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252528 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" Oct 31 15:57:46.252598 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252533 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" Oct 31 15:57:46.252729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252622 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" Oct 31 15:57:46.252729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252631 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" containerName="istio-proxy" Oct 31 15:57:46.252729 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.252643 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" containerName="revoke-keys" Oct 31 15:57:46.257667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.257650 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.311522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.311460 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:57:46.311678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.311535 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:57:46.311678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.311551 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:57:46.311678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.311595 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:57:46.316743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.316721 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj"] Oct 31 15:57:46.348947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.348911 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.349111 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.348995 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dbsj\" (UniqueName: \"kubernetes.io/projected/bb2ea294-1ecd-4d79-b1ba-63f480460945-kube-api-access-9dbsj\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.349276 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.349248 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.450252 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.450217 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.450413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.450276 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.450413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.450358 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dbsj\" (UniqueName: \"kubernetes.io/projected/bb2ea294-1ecd-4d79-b1ba-63f480460945-kube-api-access-9dbsj\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.450913 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.450881 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.452852 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.452830 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.497070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.497044 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4a93c2-2dc1-4b9a-8cf0-821630eea00d" path="/var/lib/kubelet/pods/ee4a93c2-2dc1-4b9a-8cf0-821630eea00d/volumes" Oct 31 15:57:46.508010 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.507945 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dbsj\" (UniqueName: \"kubernetes.io/projected/bb2ea294-1ecd-4d79-b1ba-63f480460945-kube-api-access-9dbsj\") pod \"maas-api-revoke-sub-8755aabf-5dtgj\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.570990 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.570960 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:46.946955 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:46.946925 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj"] Oct 31 15:57:46.948089 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:57:46.948067 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2ea294_1ecd_4d79_b1ba_63f480460945.slice/crio-a8a104b582b3657e8931ec43423a2f7914706a9f7184bf1d8abec9dd762d1184 WatchSource:0}: Error finding container a8a104b582b3657e8931ec43423a2f7914706a9f7184bf1d8abec9dd762d1184: Status 404 returned error can't find the container with id a8a104b582b3657e8931ec43423a2f7914706a9f7184bf1d8abec9dd762d1184 Oct 31 15:57:47.490236 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:47.490197 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:57:47.661049 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:47.661014 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:47.731196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:47.731164 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" event={"ID":"bb2ea294-1ecd-4d79-b1ba-63f480460945","Type":"ContainerStarted","Data":"ddc4d318d14600af09da459ef3a60e4328e45937d7d4f510565ba557fc8a5826"} Oct 31 15:57:47.731420 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:47.731213 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" event={"ID":"bb2ea294-1ecd-4d79-b1ba-63f480460945","Type":"ContainerStarted","Data":"a8a104b582b3657e8931ec43423a2f7914706a9f7184bf1d8abec9dd762d1184"} Oct 31 15:57:47.847304 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:47.847250 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" podStartSLOduration=1.847237005 podStartE2EDuration="1.847237005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:57:47.845435115 +0000 UTC m=+4313.464986527" watchObservedRunningTime="2026-10-31 15:57:47.847237005 +0000 UTC m=+4313.466788415" Oct 31 15:57:48.740023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:48.739989 4865 generic.go:356] "Generic (PLEG): container finished" podID="bb2ea294-1ecd-4d79-b1ba-63f480460945" containerID="ddc4d318d14600af09da459ef3a60e4328e45937d7d4f510565ba557fc8a5826" exitCode=0 Oct 31 15:57:48.740461 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:48.740069 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" event={"ID":"bb2ea294-1ecd-4d79-b1ba-63f480460945","Type":"ContainerDied","Data":"ddc4d318d14600af09da459ef3a60e4328e45937d7d4f510565ba557fc8a5826"} Oct 31 15:57:49.905677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:49.905653 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:49.994172 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:49.994145 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-xks-ca\") pod \"bb2ea294-1ecd-4d79-b1ba-63f480460945\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " Oct 31 15:57:49.994379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:49.994366 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-openshift-ca\") pod \"bb2ea294-1ecd-4d79-b1ba-63f480460945\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " Oct 31 15:57:49.994441 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:49.994420 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2ea294-1ecd-4d79-b1ba-63f480460945-kube-api-access-9dbsj\" (UniqueName: \"kubernetes.io/projected/bb2ea294-1ecd-4d79-b1ba-63f480460945-kube-api-access-9dbsj\") pod \"bb2ea294-1ecd-4d79-b1ba-63f480460945\" (UID: \"bb2ea294-1ecd-4d79-b1ba-63f480460945\") " Oct 31 15:57:49.994829 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:49.994796 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-openshift-ca" pod "bb2ea294-1ecd-4d79-b1ba-63f480460945" (UID: "bb2ea294-1ecd-4d79-b1ba-63f480460945"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:49.996692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:49.996659 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-xks-ca" pod "bb2ea294-1ecd-4d79-b1ba-63f480460945" (UID: "bb2ea294-1ecd-4d79-b1ba-63f480460945"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:57:49.996692 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:49.996681 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2ea294-1ecd-4d79-b1ba-63f480460945-kube-api-access-9dbsj" pod "bb2ea294-1ecd-4d79-b1ba-63f480460945" (UID: "bb2ea294-1ecd-4d79-b1ba-63f480460945"). InnerVolumeSpecName "kube-api-access-9dbsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:50.095682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:50.095639 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:50.095682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:50.095674 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ea294-1ecd-4d79-b1ba-63f480460945-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:50.095682 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:50.095689 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dbsj\" (UniqueName: \"kubernetes.io/projected/bb2ea294-1ecd-4d79-b1ba-63f480460945-kube-api-access-9dbsj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:50.414931 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:50.414841 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:50.755405 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:50.755353 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" Oct 31 15:57:50.755621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:50.755364 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj" event={"ID":"bb2ea294-1ecd-4d79-b1ba-63f480460945","Type":"ContainerDied","Data":"a8a104b582b3657e8931ec43423a2f7914706a9f7184bf1d8abec9dd762d1184"} Oct 31 15:57:50.755621 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:50.755535 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a104b582b3657e8931ec43423a2f7914706a9f7184bf1d8abec9dd762d1184" Oct 31 15:57:51.563456 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:51.563422 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" podUID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:57:51.563847 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:51.563426 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" podUID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:57:52.320369 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.320343 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:57:52.324023 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.324001 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e274d0a9-6d82-4faa-b2d7-85e3d882c094-kube-api-access-5x5bn\" (UniqueName: \"kubernetes.io/projected/e274d0a9-6d82-4faa-b2d7-85e3d882c094-kube-api-access-5x5bn\") pod \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " Oct 31 15:57:52.324144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.324124 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e274d0a9-6d82-4faa-b2d7-85e3d882c094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e274d0a9-6d82-4faa-b2d7-85e3d882c094-plugins-config-volume\") pod \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\" (UID: \"e274d0a9-6d82-4faa-b2d7-85e3d882c094\") " Oct 31 15:57:52.324464 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.324440 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e274d0a9-6d82-4faa-b2d7-85e3d882c094-plugins-config-volume" pod "e274d0a9-6d82-4faa-b2d7-85e3d882c094" (UID: "e274d0a9-6d82-4faa-b2d7-85e3d882c094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:52.324696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.324679 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e274d0a9-6d82-4faa-b2d7-85e3d882c094-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:52.326173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.326147 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e274d0a9-6d82-4faa-b2d7-85e3d882c094-kube-api-access-5x5bn" pod "e274d0a9-6d82-4faa-b2d7-85e3d882c094" (UID: "e274d0a9-6d82-4faa-b2d7-85e3d882c094"). InnerVolumeSpecName "kube-api-access-5x5bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:52.425938 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.425909 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5bn\" (UniqueName: \"kubernetes.io/projected/e274d0a9-6d82-4faa-b2d7-85e3d882c094-kube-api-access-5x5bn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:52.430613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.430589 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:57:52.527253 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.527224 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-plugins-config-volume\") pod \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " Oct 31 15:57:52.527421 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.527315 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-kube-api-access-qw6xb\" (UniqueName: \"kubernetes.io/projected/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-kube-api-access-qw6xb\") pod \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\" (UID: \"e76f37ab-69f6-4d07-8da5-ad04ce5d334b\") " Oct 31 15:57:52.527615 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.527591 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-plugins-config-volume" pod "e76f37ab-69f6-4d07-8da5-ad04ce5d334b" (UID: "e76f37ab-69f6-4d07-8da5-ad04ce5d334b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:52.527824 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.527806 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:52.529627 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.529607 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-kube-api-access-qw6xb" pod "e76f37ab-69f6-4d07-8da5-ad04ce5d334b" (UID: "e76f37ab-69f6-4d07-8da5-ad04ce5d334b"). InnerVolumeSpecName "kube-api-access-qw6xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:52.628603 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.628528 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6xb\" (UniqueName: \"kubernetes.io/projected/e76f37ab-69f6-4d07-8da5-ad04ce5d334b-kube-api-access-qw6xb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:52.773793 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.773755 4865 generic.go:356] "Generic (PLEG): container finished" podID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" containerID="70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612" exitCode=0 Oct 31 15:57:52.773980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.773799 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" Oct 31 15:57:52.773980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.773838 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" event={"ID":"e274d0a9-6d82-4faa-b2d7-85e3d882c094","Type":"ContainerDied","Data":"70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612"} Oct 31 15:57:52.773980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.773891 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl" event={"ID":"e274d0a9-6d82-4faa-b2d7-85e3d882c094","Type":"ContainerDied","Data":"2fdd8c6d437609f02da43e741f307a53b202706cbe082b88e2950e1c3093ff3f"} Oct 31 15:57:52.773980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.773913 4865 scope.go:122] "RemoveContainer" containerID="70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612" Oct 31 15:57:52.775393 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.775361 4865 generic.go:356] "Generic (PLEG): container finished" podID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" containerID="db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0" exitCode=0 Oct 31 15:57:52.775519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.775433 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" Oct 31 15:57:52.775519 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.775490 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" event={"ID":"e76f37ab-69f6-4d07-8da5-ad04ce5d334b","Type":"ContainerDied","Data":"db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0"} Oct 31 15:57:52.775610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.775523 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7" event={"ID":"e76f37ab-69f6-4d07-8da5-ad04ce5d334b","Type":"ContainerDied","Data":"80a901f4c5eb27698f4bf8d9fc1966dc003cb4232587df71a12f3aa82778dde2"} Oct 31 15:57:52.784657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.784640 4865 scope.go:122] "RemoveContainer" containerID="70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612" Oct 31 15:57:52.784921 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:52.784903 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612\": container with ID starting with 70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612 not found: ID does not exist" containerID="70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612" Oct 31 15:57:52.784975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.784928 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612"} err="failed to get container status \"70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612\": rpc error: code = NotFound desc = could not find container \"70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612\": container with ID starting with 70c4975de9573161a85eaae77ff78c3c00f8b76021621c1d23f447af43f8c612 not found: ID does not exist" Oct 31 15:57:52.784975 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.784944 4865 scope.go:122] "RemoveContainer" containerID="db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0" Oct 31 15:57:52.794169 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.794152 4865 scope.go:122] "RemoveContainer" containerID="db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0" Oct 31 15:57:52.794450 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:57:52.794431 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0\": container with ID starting with db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0 not found: ID does not exist" containerID="db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0" Oct 31 15:57:52.794528 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.794456 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0"} err="failed to get container status \"db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0\": rpc error: code = NotFound desc = could not find container \"db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0\": container with ID starting with db3bf10ec1b249ba1de999703da62e55182d2506febec5e50d955dcc08720ae0 not found: ID does not exist" Oct 31 15:57:52.940700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.940643 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7"] Oct 31 15:57:52.991509 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:52.991483 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-46bf36-5dfc994cc9-kplw7"] Oct 31 15:57:53.057780 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:53.057753 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl"] Oct 31 15:57:53.060759 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:53.060737 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-46bf36-dc56b8d69-pg8zl"] Oct 31 15:57:54.099361 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.099328 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:54.163831 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.163790 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn"] Oct 31 15:57:54.164451 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164438 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" containerName="payload-processing" Oct 31 15:57:54.164518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164453 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" containerName="payload-processing" Oct 31 15:57:54.164518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164501 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bb2ea294-1ecd-4d79-b1ba-63f480460945" containerName="revoke-keys" Oct 31 15:57:54.164518 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164509 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2ea294-1ecd-4d79-b1ba-63f480460945" containerName="revoke-keys" Oct 31 15:57:54.164620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164530 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" containerName="payload-pre-processing" Oct 31 15:57:54.164620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164538 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" containerName="payload-pre-processing" Oct 31 15:57:54.164694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164667 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" containerName="payload-processing" Oct 31 15:57:54.164694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164680 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2ea294-1ecd-4d79-b1ba-63f480460945" containerName="revoke-keys" Oct 31 15:57:54.164694 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.164689 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" containerName="payload-pre-processing" Oct 31 15:57:54.169641 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.169625 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.225372 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.225343 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn"] Oct 31 15:57:54.248463 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.248435 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.248688 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.248668 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p8nn\" (UniqueName: \"kubernetes.io/projected/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-kube-api-access-9p8nn\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.248853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.248826 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.270908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.270884 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:57:54.270908 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.270887 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 15:57:54.271112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.271046 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 15:57:54.273657 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.273640 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:57:54.350121 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.350043 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p8nn\" (UniqueName: \"kubernetes.io/projected/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-kube-api-access-9p8nn\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.350270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.350161 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.350270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.350239 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.350948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.350903 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.352677 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.352656 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.407606 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.407573 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p8nn\" (UniqueName: \"kubernetes.io/projected/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-kube-api-access-9p8nn\") pod \"maas-api-revoke-sub-c468c833-9j8qn\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.483577 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.483547 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:54.509363 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.509334 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e274d0a9-6d82-4faa-b2d7-85e3d882c094" path="/var/lib/kubelet/pods/e274d0a9-6d82-4faa-b2d7-85e3d882c094/volumes" Oct 31 15:57:54.509879 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.509857 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76f37ab-69f6-4d07-8da5-ad04ce5d334b" path="/var/lib/kubelet/pods/e76f37ab-69f6-4d07-8da5-ad04ce5d334b/volumes" Oct 31 15:57:54.516045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.516018 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:54.639846 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.639742 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:54.867818 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:54.867796 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn"] Oct 31 15:57:54.868515 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:57:54.868451 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d238adf_64a6_4e2d_9872_aa7d0ffa54fa.slice/crio-bfc02bd4afbae9b732ef1490303a014042598c3645d48575f96867510516d7e6 WatchSource:0}: Error finding container bfc02bd4afbae9b732ef1490303a014042598c3645d48575f96867510516d7e6: Status 404 returned error can't find the container with id bfc02bd4afbae9b732ef1490303a014042598c3645d48575f96867510516d7e6 Oct 31 15:57:55.164722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:55.164644 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:55.800037 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:55.800008 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" event={"ID":"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa","Type":"ContainerStarted","Data":"38112de63ab1c6433d50d8145eaa81df7783555469e5406c6e2d9c66579849ee"} Oct 31 15:57:55.800180 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:55.800066 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" event={"ID":"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa","Type":"ContainerStarted","Data":"bfc02bd4afbae9b732ef1490303a014042598c3645d48575f96867510516d7e6"} Oct 31 15:57:55.920999 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:55.920942 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" podStartSLOduration=1.920928531 podStartE2EDuration="1.920928531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:57:55.918434008 +0000 UTC m=+4321.537985420" watchObservedRunningTime="2026-10-31 15:57:55.920928531 +0000 UTC m=+4321.540479940" Oct 31 15:57:56.808751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:56.808710 4865 generic.go:356] "Generic (PLEG): container finished" podID="8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" containerID="38112de63ab1c6433d50d8145eaa81df7783555469e5406c6e2d9c66579849ee" exitCode=0 Oct 31 15:57:56.809223 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:56.808794 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" event={"ID":"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa","Type":"ContainerDied","Data":"38112de63ab1c6433d50d8145eaa81df7783555469e5406c6e2d9c66579849ee"} Oct 31 15:57:56.972602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:56.972573 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:57:57.970006 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:57.969985 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:58.008579 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.008554 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-xks-ca\") pod \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " Oct 31 15:57:58.008755 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.008742 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-openshift-ca\") pod \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " Oct 31 15:57:58.008803 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.008784 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-kube-api-access-9p8nn\" (UniqueName: \"kubernetes.io/projected/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-kube-api-access-9p8nn\") pod \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\" (UID: \"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa\") " Oct 31 15:57:58.009088 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.009063 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-openshift-ca" pod "8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" (UID: "8d238adf-64a6-4e2d-9872-aa7d0ffa54fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:58.010732 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.010708 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-xks-ca" pod "8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" (UID: "8d238adf-64a6-4e2d-9872-aa7d0ffa54fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:57:58.010966 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.010941 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-kube-api-access-9p8nn" pod "8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" (UID: "8d238adf-64a6-4e2d-9872-aa7d0ffa54fa"). InnerVolumeSpecName "kube-api-access-9p8nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:58.110118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.110048 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p8nn\" (UniqueName: \"kubernetes.io/projected/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-kube-api-access-9p8nn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:58.110118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.110078 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:58.110118 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.110090 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:58.823495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.823394 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" Oct 31 15:57:58.823495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.823401 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn" event={"ID":"8d238adf-64a6-4e2d-9872-aa7d0ffa54fa","Type":"ContainerDied","Data":"bfc02bd4afbae9b732ef1490303a014042598c3645d48575f96867510516d7e6"} Oct 31 15:57:58.823495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:58.823443 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfc02bd4afbae9b732ef1490303a014042598c3645d48575f96867510516d7e6" Oct 31 15:57:59.647959 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:57:59.647923 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:00.017070 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.017040 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:00.382447 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.382367 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" containerID="cri-o://c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e" gracePeriod=45 Oct 31 15:58:00.519984 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.519952 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": dial tcp 10.132.0.89:8000: connect: connection refused" Oct 31 15:58:00.714611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.714588 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:58:00.841359 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.841327 4865 generic.go:356] "Generic (PLEG): container finished" podID="bab3d202-010d-439a-b663-e79536568536" containerID="c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e" exitCode=0 Oct 31 15:58:00.841574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.841396 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" event={"ID":"bab3d202-010d-439a-b663-e79536568536","Type":"ContainerDied","Data":"c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e"} Oct 31 15:58:00.841574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.841434 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" Oct 31 15:58:00.841574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.841447 4865 scope.go:122] "RemoveContainer" containerID="c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e" Oct 31 15:58:00.841574 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.841434 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw" event={"ID":"bab3d202-010d-439a-b663-e79536568536","Type":"ContainerDied","Data":"0839ad87f13f31134057c30dacc681546e011fd7dd9191d1257f9e0954e9c51f"} Oct 31 15:58:00.844083 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844054 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-dshm\") pod \"bab3d202-010d-439a-b663-e79536568536\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " Oct 31 15:58:00.844226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844108 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-tmp-dir\") pod \"bab3d202-010d-439a-b663-e79536568536\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " Oct 31 15:58:00.844226 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844144 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-model-cache\") pod \"bab3d202-010d-439a-b663-e79536568536\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " Oct 31 15:58:00.844358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844305 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab3d202-010d-439a-b663-e79536568536-kube-api-access-44g9s\" (UniqueName: \"kubernetes.io/projected/bab3d202-010d-439a-b663-e79536568536-kube-api-access-44g9s\") pod \"bab3d202-010d-439a-b663-e79536568536\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " Oct 31 15:58:00.844358 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844335 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab3d202-010d-439a-b663-e79536568536-tls-certs\" (UniqueName: \"kubernetes.io/secret/bab3d202-010d-439a-b663-e79536568536-tls-certs\") pod \"bab3d202-010d-439a-b663-e79536568536\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " Oct 31 15:58:00.844500 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844440 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-tmp-dir" pod "bab3d202-010d-439a-b663-e79536568536" (UID: "bab3d202-010d-439a-b663-e79536568536"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:58:00.844556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844516 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-model-cache" pod "bab3d202-010d-439a-b663-e79536568536" (UID: "bab3d202-010d-439a-b663-e79536568536"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:58:00.844623 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.844602 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-home\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-home\") pod \"bab3d202-010d-439a-b663-e79536568536\" (UID: \"bab3d202-010d-439a-b663-e79536568536\") " Oct 31 15:58:00.845164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.845137 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:00.845271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.845169 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:00.845271 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.845259 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-home" pod "bab3d202-010d-439a-b663-e79536568536" (UID: "bab3d202-010d-439a-b663-e79536568536"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:58:00.847961 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.847933 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab3d202-010d-439a-b663-e79536568536-kube-api-access-44g9s" pod "bab3d202-010d-439a-b663-e79536568536" (UID: "bab3d202-010d-439a-b663-e79536568536"). InnerVolumeSpecName "kube-api-access-44g9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:58:00.848513 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.848487 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab3d202-010d-439a-b663-e79536568536-tls-certs" pod "bab3d202-010d-439a-b663-e79536568536" (UID: "bab3d202-010d-439a-b663-e79536568536"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:58:00.848741 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.848724 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-dshm" pod "bab3d202-010d-439a-b663-e79536568536" (UID: "bab3d202-010d-439a-b663-e79536568536"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:58:00.876605 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.876577 4865 scope.go:122] "RemoveContainer" containerID="c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e" Oct 31 15:58:00.877056 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:58:00.876989 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e\": container with ID starting with c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e not found: ID does not exist" containerID="c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e" Oct 31 15:58:00.877056 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.877028 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e"} err="failed to get container status \"c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e\": rpc error: code = NotFound desc = could not find container \"c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e\": container with ID starting with c11edc776e450bfb3dd8e4520bcf832cb07f0c58b3540e987cb154d05ef4e89e not found: ID does not exist" Oct 31 15:58:00.946330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.946223 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:00.946330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.946276 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44g9s\" (UniqueName: \"kubernetes.io/projected/bab3d202-010d-439a-b663-e79536568536-kube-api-access-44g9s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:00.946330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.946307 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bab3d202-010d-439a-b663-e79536568536-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:00.946330 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:00.946316 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bab3d202-010d-439a-b663-e79536568536-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:01.266427 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:01.266393 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw"] Oct 31 15:58:01.317556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:01.317530 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-2-w4-57815b-kserve-6c4c44df66-tpnmw"] Oct 31 15:58:02.497637 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:02.497606 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab3d202-010d-439a-b663-e79536568536" path="/var/lib/kubelet/pods/bab3d202-010d-439a-b663-e79536568536/volumes" Oct 31 15:58:03.619453 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:03.619406 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:04.815495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:04.813828 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:05.586091 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:05.586053 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:09.491535 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:09.491499 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:09.699302 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:09.699261 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:11.251764 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:11.251733 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:11.925035 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:11.925003 4865 generic.go:356] "Generic (PLEG): container finished" podID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerID="cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97" exitCode=0 Oct 31 15:58:11.925673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:11.925085 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" event={"ID":"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21","Type":"ContainerDied","Data":"cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97"} Oct 31 15:58:11.925673 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:11.925584 4865 scope.go:122] "RemoveContainer" containerID="cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97" Oct 31 15:58:11.927144 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:11.927109 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:58:11.927243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:11.927195 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:58:11.927283 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:11.927237 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:58:12.934400 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:12.934363 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" event={"ID":"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21","Type":"ContainerStarted","Data":"d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce"} Oct 31 15:58:13.633310 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:13.633272 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:58:13.633543 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:13.633515 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 31 15:58:13.633610 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:13.633566 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 31 15:58:14.190007 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:14.189970 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:14.633332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:14.633295 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 31 15:58:14.633531 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:14.633352 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 31 15:58:15.633278 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:15.633245 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 31 15:58:15.633730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:15.633321 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 31 15:58:16.633317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:16.633283 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 31 15:58:16.633801 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:16.633339 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 31 15:58:17.490986 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:58:17.490938 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:58:17.633388 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:17.633353 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" start-of-body= Oct 31 15:58:17.633995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:17.633429 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.99:15021/healthz/ready\": dial tcp 10.132.1.99:15021: connect: connection refused" Oct 31 15:58:18.638706 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:18.638674 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:58:18.639216 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:18.639139 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:58:18.639997 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:18.639976 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 15:58:18.825855 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:18.825820 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:19.742702 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:19.742668 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:22.147277 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:22.147245 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:22.269498 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:22.269435 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:23.515171 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:23.515122 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:23.669040 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:23.668998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:25.620510 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:25.620463 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:26.093992 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:26.093954 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx"] Oct 31 15:58:26.158217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:26.158186 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5"] Oct 31 15:58:26.399701 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:26.399622 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:26.524724 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:26.524696 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:28.468350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:28.468305 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:29.490944 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:29.490908 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:29.611722 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:29.611690 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:31.094454 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.094409 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" podUID="c6938d10-b7d4-4e0e-bc23-849e1ca85452" containerName="payload-processing" containerID="cri-o://61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390" gracePeriod=25 Oct 31 15:58:31.158980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.158939 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" podUID="f3ad7d6e-214b-45d7-ad3e-9e33b43af308" containerName="payload-pre-processing" containerID="cri-o://1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606" gracePeriod=25 Oct 31 15:58:31.431601 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.431566 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:58:31.468675 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.468650 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:58:31.544514 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.544482 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:58:31.584287 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.584243 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6938d10-b7d4-4e0e-bc23-849e1ca85452-kube-api-access-6mqxj\" (UniqueName: \"kubernetes.io/projected/c6938d10-b7d4-4e0e-bc23-849e1ca85452-kube-api-access-6mqxj\") pod \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " Oct 31 15:58:31.584538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.584447 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6938d10-b7d4-4e0e-bc23-849e1ca85452-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6938d10-b7d4-4e0e-bc23-849e1ca85452-plugins-config-volume\") pod \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\" (UID: \"c6938d10-b7d4-4e0e-bc23-849e1ca85452\") " Oct 31 15:58:31.584628 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.584593 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-kube-api-access-s944q\" (UniqueName: \"kubernetes.io/projected/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-kube-api-access-s944q\") pod \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " Oct 31 15:58:31.584810 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.584787 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-plugins-config-volume\") pod \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\" (UID: \"f3ad7d6e-214b-45d7-ad3e-9e33b43af308\") " Oct 31 15:58:31.584894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.584830 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6938d10-b7d4-4e0e-bc23-849e1ca85452-plugins-config-volume" pod "c6938d10-b7d4-4e0e-bc23-849e1ca85452" (UID: "c6938d10-b7d4-4e0e-bc23-849e1ca85452"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:58:31.585134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.585110 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-plugins-config-volume" pod "f3ad7d6e-214b-45d7-ad3e-9e33b43af308" (UID: "f3ad7d6e-214b-45d7-ad3e-9e33b43af308"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:58:31.586275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.586093 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6938d10-b7d4-4e0e-bc23-849e1ca85452-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:31.586275 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.586214 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:31.586894 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.586875 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6938d10-b7d4-4e0e-bc23-849e1ca85452-kube-api-access-6mqxj" pod "c6938d10-b7d4-4e0e-bc23-849e1ca85452" (UID: "c6938d10-b7d4-4e0e-bc23-849e1ca85452"). InnerVolumeSpecName "kube-api-access-6mqxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:58:31.587112 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.587092 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-kube-api-access-s944q" pod "f3ad7d6e-214b-45d7-ad3e-9e33b43af308" (UID: "f3ad7d6e-214b-45d7-ad3e-9e33b43af308"). InnerVolumeSpecName "kube-api-access-s944q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:58:31.688167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.688061 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqxj\" (UniqueName: \"kubernetes.io/projected/c6938d10-b7d4-4e0e-bc23-849e1ca85452-kube-api-access-6mqxj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:31.688167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:31.688112 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s944q\" (UniqueName: \"kubernetes.io/projected/f3ad7d6e-214b-45d7-ad3e-9e33b43af308-kube-api-access-s944q\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:32.065225 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.065182 4865 generic.go:356] "Generic (PLEG): container finished" podID="f3ad7d6e-214b-45d7-ad3e-9e33b43af308" containerID="1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606" exitCode=0 Oct 31 15:58:32.065413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.065259 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" Oct 31 15:58:32.065413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.065280 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" event={"ID":"f3ad7d6e-214b-45d7-ad3e-9e33b43af308","Type":"ContainerDied","Data":"1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606"} Oct 31 15:58:32.065413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.065355 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5" event={"ID":"f3ad7d6e-214b-45d7-ad3e-9e33b43af308","Type":"ContainerDied","Data":"656516dd0a1b9988bac20ffb70f109760390a437ac183e743c8277c99d834328"} Oct 31 15:58:32.065413 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.065386 4865 scope.go:122] "RemoveContainer" containerID="1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606" Oct 31 15:58:32.066949 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.066927 4865 generic.go:356] "Generic (PLEG): container finished" podID="c6938d10-b7d4-4e0e-bc23-849e1ca85452" containerID="61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390" exitCode=0 Oct 31 15:58:32.067053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.066985 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" Oct 31 15:58:32.067053 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.067015 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" event={"ID":"c6938d10-b7d4-4e0e-bc23-849e1ca85452","Type":"ContainerDied","Data":"61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390"} Oct 31 15:58:32.067163 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.067069 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx" event={"ID":"c6938d10-b7d4-4e0e-bc23-849e1ca85452","Type":"ContainerDied","Data":"8a5233784b96c8dd511d2104111a2f4d5d05475237bf3683147ec779b244cf73"} Oct 31 15:58:32.076529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.076512 4865 scope.go:122] "RemoveContainer" containerID="1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606" Oct 31 15:58:32.076848 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:58:32.076823 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606\": container with ID starting with 1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606 not found: ID does not exist" containerID="1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606" Oct 31 15:58:32.076923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.076857 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606"} err="failed to get container status \"1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606\": rpc error: code = NotFound desc = could not find container \"1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606\": container with ID starting with 1458123d4f47fa6da6cc56d94d22d2f13ceef104ef07ef962ae7b9af2dff4606 not found: ID does not exist" Oct 31 15:58:32.076923 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.076874 4865 scope.go:122] "RemoveContainer" containerID="61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390" Oct 31 15:58:32.087381 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.087359 4865 scope.go:122] "RemoveContainer" containerID="61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390" Oct 31 15:58:32.087730 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:58:32.087709 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390\": container with ID starting with 61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390 not found: ID does not exist" containerID="61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390" Oct 31 15:58:32.087785 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.087744 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390"} err="failed to get container status \"61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390\": rpc error: code = NotFound desc = could not find container \"61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390\": container with ID starting with 61989365fd5d8338cd6e8f2edce5b75f1a98277e3abf12feb1e291070c72d390 not found: ID does not exist" Oct 31 15:58:32.131346 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.131316 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:32.232896 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.232856 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx"] Oct 31 15:58:32.235905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.235874 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9d627707-6985d4f867-wvbjx"] Oct 31 15:58:32.345314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.345230 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5"] Oct 31 15:58:32.348444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.348416 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d627707-74895764d7-tdtd5"] Oct 31 15:58:32.496947 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.496910 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6938d10-b7d4-4e0e-bc23-849e1ca85452" path="/var/lib/kubelet/pods/c6938d10-b7d4-4e0e-bc23-849e1ca85452/volumes" Oct 31 15:58:32.497416 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.497393 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ad7d6e-214b-45d7-ad3e-9e33b43af308" path="/var/lib/kubelet/pods/f3ad7d6e-214b-45d7-ad3e-9e33b43af308/volumes" Oct 31 15:58:32.864239 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.864202 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c"] Oct 31 15:58:32.869494 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.865444 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" Oct 31 15:58:32.869704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.869683 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" Oct 31 15:58:32.869873 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.869857 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c6938d10-b7d4-4e0e-bc23-849e1ca85452" containerName="payload-processing" Oct 31 15:58:32.869982 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.869968 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6938d10-b7d4-4e0e-bc23-849e1ca85452" containerName="payload-processing" Oct 31 15:58:32.870101 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.870089 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" containerName="revoke-keys" Oct 31 15:58:32.870192 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.870181 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" containerName="revoke-keys" Oct 31 15:58:32.870305 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.870294 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f3ad7d6e-214b-45d7-ad3e-9e33b43af308" containerName="payload-pre-processing" Oct 31 15:58:32.870395 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.870381 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ad7d6e-214b-45d7-ad3e-9e33b43af308" containerName="payload-pre-processing" Oct 31 15:58:32.870713 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.870694 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" containerName="revoke-keys" Oct 31 15:58:32.870843 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.870829 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ad7d6e-214b-45d7-ad3e-9e33b43af308" containerName="payload-pre-processing" Oct 31 15:58:32.870956 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.870944 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab3d202-010d-439a-b663-e79536568536" containerName="main" Oct 31 15:58:32.871063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.871050 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6938d10-b7d4-4e0e-bc23-849e1ca85452" containerName="payload-processing" Oct 31 15:58:32.891744 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.891718 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.922644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.922534 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.922958 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.922905 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.923273 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.923204 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f75e4-410a-42c9-8908-32f009e19525-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.923488 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.923412 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f75e4-410a-42c9-8908-32f009e19525-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.923745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.923626 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.923745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.923696 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.924105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.923933 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.924105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.924011 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.924105 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.924042 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9b8x\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-kube-api-access-f9b8x\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:32.947907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:32.947875 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-57815b-openshift-default-dockercfg-9dqr9\"" Oct 31 15:58:33.089875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.062752 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f75e4-410a-42c9-8908-32f009e19525-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.108195 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.108163 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f75e4-410a-42c9-8908-32f009e19525-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.154877 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.154744 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.156084 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.156057 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.263784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.263748 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.293355 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.293321 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.294177 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.294157 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9b8x\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-kube-api-access-f9b8x\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.335918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.292918 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.335918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.163739 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.335918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.294098 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f75e4-410a-42c9-8908-32f009e19525-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.335918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.294071 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.335918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.318919 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.335918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.319407 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.368876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.356869 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.401980 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.401087 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.419208 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.416285 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c"] Oct 31 15:58:33.421244 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.421177 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9b8x\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-kube-api-access-f9b8x\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.435918 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.435845 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f75e4-410a-42c9-8908-32f009e19525-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.438533 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.438453 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.562024 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.561723 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:33.665444 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.664887 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:33.980020 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:33.979983 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:34.113669 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.112998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:34.170620 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.170564 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:58:34.351554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.305311 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:58:34.351554 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.319142 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:58:34.673131 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.673026 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 15:58:34.711505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.710498 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 15:58:34.768028 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.767963 4865 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:58:34.819699 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:58:34.819647 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00f75e4_410a_42c9_8908_32f009e19525.slice/crio-efe19f8b27d4881496f2e3017da5f06d895b6b3cfc596e8bbaaf2c6070e20e8e WatchSource:0}: Error finding container efe19f8b27d4881496f2e3017da5f06d895b6b3cfc596e8bbaaf2c6070e20e8e: Status 404 returned error can't find the container with id efe19f8b27d4881496f2e3017da5f06d895b6b3cfc596e8bbaaf2c6070e20e8e Oct 31 15:58:34.858512 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.831186 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c"] Oct 31 15:58:34.875940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.875892 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:58:34.876270 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.876244 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:58:34.876501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:34.876451 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 15:58:35.198622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.198570 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.112:8000/ready\": context deadline exceeded" Oct 31 15:58:35.199828 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.199806 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:58:35.317758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.317714 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Oct 31 15:58:35.319751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.319723 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:58:35.321288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.320410 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:58:35.324589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.324562 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:58:35.355720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.355661 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:58:35.593644 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.593554 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 15:58:35.601082 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.596890 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 15:58:35.629375 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.629272 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 15:58:35.870541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.870435 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" event={"ID":"a00f75e4-410a-42c9-8908-32f009e19525","Type":"ContainerStarted","Data":"fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942"} Oct 31 15:58:35.870541 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:35.870527 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" event={"ID":"a00f75e4-410a-42c9-8908-32f009e19525","Type":"ContainerStarted","Data":"efe19f8b27d4881496f2e3017da5f06d895b6b3cfc596e8bbaaf2c6070e20e8e"} Oct 31 15:58:36.125745 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:36.125643 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podStartSLOduration=4.125626702 podStartE2EDuration="4.125626702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:58:36.122592768 +0000 UTC m=+4361.742144180" watchObservedRunningTime="2026-10-31 15:58:36.125626702 +0000 UTC m=+4361.745178113" Oct 31 15:58:36.563250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:36.563211 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:36.564589 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:36.564562 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" start-of-body= Oct 31 15:58:36.564717 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:36.564613 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" Oct 31 15:58:37.562749 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:37.562709 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" start-of-body= Oct 31 15:58:37.562967 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:37.562780 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" Oct 31 15:58:38.408495 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:38.406986 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:38.562907 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:38.562869 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" start-of-body= Oct 31 15:58:38.563098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:38.562927 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" Oct 31 15:58:39.563426 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:39.563388 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" start-of-body= Oct 31 15:58:39.563870 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:39.563446 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" Oct 31 15:58:40.563063 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:40.563032 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" start-of-body= Oct 31 15:58:40.563251 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:40.563083 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" Oct 31 15:58:41.562757 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:41.562723 4865 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" start-of-body= Oct 31 15:58:41.563198 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:41.562783 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.103:15021/healthz/ready\": dial tcp 10.132.1.103:15021: connect: connection refused" Oct 31 15:58:42.567885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:42.567847 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:42.568344 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:42.568331 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:42.569243 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:42.569223 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:58:43.201373 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:43.201334 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:47.500275 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:58:47.500236 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:58:47.515391 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:47.515358 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:48.513626 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:48.513595 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:48.915148 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:48.915067 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:49.232410 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:49.232378 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:49.500905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:49.500820 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:49.765643 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:49.765559 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:53.525631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:53.525593 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:53.909570 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:53.905240 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:54.048993 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:54.048956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:54.180098 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:54.180016 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:55.460622 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:55.460590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:55.901720 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:55.901637 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:56.031173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:56.031141 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:57.040220 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.040174 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:57.216905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.216857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:57.470129 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.470094 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl"] Oct 31 15:58:57.474515 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.474463 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:57.531014 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.530979 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl"] Oct 31 15:58:57.651678 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.617236 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-140c48-dockercfg-6nrcn\"" Oct 31 15:58:57.651940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.622436 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-140c48\"" Oct 31 15:58:57.651940 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.625010 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gs89\" (UniqueName: \"kubernetes.io/projected/e2e0e1ff-f21e-4725-b0c6-a523bb319651-kube-api-access-7gs89\") pod \"payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:57.652164 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.652000 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e0e1ff-f21e-4725-b0c6-a523bb319651-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:57.753830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.753743 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gs89\" (UniqueName: \"kubernetes.io/projected/e2e0e1ff-f21e-4725-b0c6-a523bb319651-kube-api-access-7gs89\") pod \"payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:57.753830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.753793 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e0e1ff-f21e-4725-b0c6-a523bb319651-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:57.754569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.754550 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e0e1ff-f21e-4725-b0c6-a523bb319651-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:57.811853 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:57.811821 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gs89\" (UniqueName: \"kubernetes.io/projected/e2e0e1ff-f21e-4725-b0c6-a523bb319651-kube-api-access-7gs89\") pod \"payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:58.088445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:58.088358 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:58.092102 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:58.092077 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:58.464382 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:58.464354 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl"] Oct 31 15:58:58.464696 ip-10-0-131-60 kubenswrapper[4865]: W1031 15:58:58.464671 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e0e1ff_f21e_4725_b0c6_a523bb319651.slice/crio-5de4605f5015ed624474b8c928a573091e020335eb64b0b3ce64da6b973cc834 WatchSource:0}: Error finding container 5de4605f5015ed624474b8c928a573091e020335eb64b0b3ce64da6b973cc834: Status 404 returned error can't find the container with id 5de4605f5015ed624474b8c928a573091e020335eb64b0b3ce64da6b973cc834 Oct 31 15:58:58.563891 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:58.563859 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:58.565647 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:58.565627 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c"] Oct 31 15:58:58.565812 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:58.565795 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" containerID="cri-o://fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942" gracePeriod=30 Oct 31 15:58:59.072406 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:59.072368 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" event={"ID":"e2e0e1ff-f21e-4725-b0c6-a523bb319651","Type":"ContainerStarted","Data":"e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013"} Oct 31 15:58:59.072665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:59.072424 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" event={"ID":"e2e0e1ff-f21e-4725-b0c6-a523bb319651","Type":"ContainerStarted","Data":"5de4605f5015ed624474b8c928a573091e020335eb64b0b3ce64da6b973cc834"} Oct 31 15:58:59.072665 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:59.072509 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:58:59.190743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:59.190692 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" podStartSLOduration=2.190677959 podStartE2EDuration="2.190677959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:58:59.189962227 +0000 UTC m=+4384.809513638" watchObservedRunningTime="2026-10-31 15:58:59.190677959 +0000 UTC m=+4384.810229370" Oct 31 15:58:59.483671 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:59.483636 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:58:59.748324 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:59.748242 4865 scope.go:122] "RemoveContainer" containerID="e53ef43a4b4a5075fe32a0944e6b2081989fe00929f505ea585bb62b7063ecbc" Oct 31 15:58:59.758920 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:58:59.758895 4865 scope.go:122] "RemoveContainer" containerID="11d7c4b499d49d3b90bc44e2b222d174c49dda73a45fe63d0829899b0a57f34f" Oct 31 15:59:00.020911 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:00.020827 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj"] Oct 31 15:59:00.021146 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:00.021117 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" containerID="cri-o://48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743" gracePeriod=30 Oct 31 15:59:00.029133 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:00.029099 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:00.445134 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:00.445046 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:00.580854 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:00.580822 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:00.725655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:00.725609 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:00.872667 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:00.872626 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:01.000206 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:01.000122 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:02.724093 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:02.724053 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:02.847973 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:02.847934 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:03.855660 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:03.855635 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:59:04.020935 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.020901 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-certs\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.020941 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-envoy\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.020992 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a00f75e4-410a-42c9-8908-32f009e19525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f75e4-410a-42c9-8908-32f009e19525-istio-podinfo\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021127 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021052 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-credential-socket\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021131 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-kube-api-access-f9b8x\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-kube-api-access-f9b8x\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021183 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-data\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021211 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00f75e4-410a-42c9-8908-32f009e19525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f75e4-410a-42c9-8908-32f009e19525-istiod-ca-cert\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021228 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-certs" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.021315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021252 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-socket\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021315 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021303 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-istio-token\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-istio-token\") pod \"a00f75e4-410a-42c9-8908-32f009e19525\" (UID: \"a00f75e4-410a-42c9-8908-32f009e19525\") " Oct 31 15:59:04.021705 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021582 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-socket" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.022286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021778 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-data" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.022286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.021866 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00f75e4-410a-42c9-8908-32f009e19525-istiod-ca-cert" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:59:04.022286 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.022073 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-credential-socket" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.022576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.022356 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.022576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.022379 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.022576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.022394 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.022576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.022410 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a00f75e4-410a-42c9-8908-32f009e19525-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.022576 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.022425 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.023941 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.023911 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-envoy" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.024067 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.024030 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-kube-api-access-f9b8x" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "kube-api-access-f9b8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:04.024340 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.024316 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a00f75e4-410a-42c9-8908-32f009e19525-istio-podinfo" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:59:04.024611 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.024582 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-istio-token" pod "a00f75e4-410a-42c9-8908-32f009e19525" (UID: "a00f75e4-410a-42c9-8908-32f009e19525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:04.107649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.107554 4865 generic.go:356] "Generic (PLEG): container finished" podID="a00f75e4-410a-42c9-8908-32f009e19525" containerID="fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942" exitCode=0 Oct 31 15:59:04.107649 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.107620 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" Oct 31 15:59:04.107876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.107643 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" event={"ID":"a00f75e4-410a-42c9-8908-32f009e19525","Type":"ContainerDied","Data":"fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942"} Oct 31 15:59:04.107876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.107725 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c" event={"ID":"a00f75e4-410a-42c9-8908-32f009e19525","Type":"ContainerDied","Data":"efe19f8b27d4881496f2e3017da5f06d895b6b3cfc596e8bbaaf2c6070e20e8e"} Oct 31 15:59:04.107876 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.107748 4865 scope.go:122] "RemoveContainer" containerID="fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942" Oct 31 15:59:04.118885 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.118863 4865 scope.go:122] "RemoveContainer" containerID="fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942" Oct 31 15:59:04.119178 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:59:04.119155 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942\": container with ID starting with fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942 not found: ID does not exist" containerID="fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942" Oct 31 15:59:04.119245 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.119192 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942"} err="failed to get container status \"fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942\": rpc error: code = NotFound desc = could not find container \"fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942\": container with ID starting with fa813f53c51edac5c4c301e3c56e93ada999f19f0e3735e97129d06841e13942 not found: ID does not exist" Oct 31 15:59:04.124100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.124025 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a00f75e4-410a-42c9-8908-32f009e19525-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.124100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.124061 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a00f75e4-410a-42c9-8908-32f009e19525-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.124100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.124078 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9b8x\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-kube-api-access-f9b8x\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.124100 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.124094 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00f75e4-410a-42c9-8908-32f009e19525-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.275948 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.275916 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c"] Oct 31 15:59:04.278875 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.278854 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-sw68c"] Oct 31 15:59:04.497608 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:04.497579 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00f75e4-410a-42c9-8908-32f009e19525" path="/var/lib/kubelet/pods/a00f75e4-410a-42c9-8908-32f009e19525/volumes" Oct 31 15:59:05.306529 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.306495 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:59:05.439031 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.438992 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-data\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439055 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-socket\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439090 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-kube-api-access-9bwpp\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-kube-api-access-9bwpp\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439126 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-envoy\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439181 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b670827b-dd4e-4e9f-938b-194b28436904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b670827b-dd4e-4e9f-938b-194b28436904-istiod-ca-cert\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439207 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-certs\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439250 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439250 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b670827b-dd4e-4e9f-938b-194b28436904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b670827b-dd4e-4e9f-938b-194b28436904-istio-podinfo\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439308 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-credential-socket\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439304 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-data" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:05.439650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439356 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-socket" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:05.439650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439440 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-istio-token\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-istio-token\") pod \"b670827b-dd4e-4e9f-938b-194b28436904\" (UID: \"b670827b-dd4e-4e9f-938b-194b28436904\") " Oct 31 15:59:05.439650 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439622 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-certs" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:05.439926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439677 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-credential-socket" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:05.439926 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439714 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b670827b-dd4e-4e9f-938b-194b28436904-istiod-ca-cert" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:59:05.440200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.439998 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.440200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.440023 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.440200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.440036 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.440200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.440049 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b670827b-dd4e-4e9f-938b-194b28436904-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.440200 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.440063 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.441807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.441774 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-kube-api-access-9bwpp" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "kube-api-access-9bwpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:05.441807 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.441774 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-envoy" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:05.442036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.441822 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b670827b-dd4e-4e9f-938b-194b28436904-istio-podinfo" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:59:05.442036 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.441931 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-istio-token" pod "b670827b-dd4e-4e9f-938b-194b28436904" (UID: "b670827b-dd4e-4e9f-938b-194b28436904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:05.541288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.541193 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bwpp\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-kube-api-access-9bwpp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.541288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.541230 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b670827b-dd4e-4e9f-938b-194b28436904-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.541288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.541241 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b670827b-dd4e-4e9f-938b-194b28436904-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.541288 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.541250 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b670827b-dd4e-4e9f-938b-194b28436904-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:05.790700 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:05.790664 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:06.124586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.124550 4865 generic.go:356] "Generic (PLEG): container finished" podID="b670827b-dd4e-4e9f-938b-194b28436904" containerID="48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743" exitCode=0 Oct 31 15:59:06.124586 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.124588 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" event={"ID":"b670827b-dd4e-4e9f-938b-194b28436904","Type":"ContainerDied","Data":"48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743"} Oct 31 15:59:06.124842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.124606 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" Oct 31 15:59:06.124842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.124631 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj" event={"ID":"b670827b-dd4e-4e9f-938b-194b28436904","Type":"ContainerDied","Data":"c79c8b232f124ccf9b90e16c468f00f492dadcc852317e7499e8e1fcd0e0e172"} Oct 31 15:59:06.124842 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.124652 4865 scope.go:122] "RemoveContainer" containerID="48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743" Oct 31 15:59:06.136664 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.136569 4865 scope.go:122] "RemoveContainer" containerID="de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9" Oct 31 15:59:06.147086 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.147063 4865 scope.go:122] "RemoveContainer" containerID="48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743" Oct 31 15:59:06.147393 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:59:06.147369 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743\": container with ID starting with 48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743 not found: ID does not exist" containerID="48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743" Oct 31 15:59:06.147485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.147402 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743"} err="failed to get container status \"48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743\": rpc error: code = NotFound desc = could not find container \"48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743\": container with ID starting with 48f601f6c0b81f9c377992aa282b235248a7c4a688cb52b000fc0134c3f85743 not found: ID does not exist" Oct 31 15:59:06.147485 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.147420 4865 scope.go:122] "RemoveContainer" containerID="de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9" Oct 31 15:59:06.147749 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:59:06.147729 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9\": container with ID starting with de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9 not found: ID does not exist" containerID="de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9" Oct 31 15:59:06.147799 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.147755 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9"} err="failed to get container status \"de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9\": rpc error: code = NotFound desc = could not find container \"de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9\": container with ID starting with de31e3b8fefc0322ce9cc5a754adb43a1501c3ecd874140b86d2049800ff18d9 not found: ID does not exist" Oct 31 15:59:06.299501 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.299454 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj"] Oct 31 15:59:06.350521 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.350452 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9d627707-openshift-default-646d4d598f-hqpvj"] Oct 31 15:59:06.498968 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.498928 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b670827b-dd4e-4e9f-938b-194b28436904" path="/var/lib/kubelet/pods/b670827b-dd4e-4e9f-938b-194b28436904/volumes" Oct 31 15:59:06.887553 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:06.887410 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:07.012936 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:07.012900 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:07.301538 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:07.301502 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:10.038797 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:10.038766 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:10.082631 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:10.082603 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 15:59:10.504964 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:10.504932 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:11.548888 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:11.548845 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:13.076946 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:13.076911 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:13.811928 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:13.811885 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4"] Oct 31 15:59:13.921167 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:13.921136 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl"] Oct 31 15:59:14.158329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:14.158227 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:14.283445 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:14.283413 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:15.410624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:15.410593 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:16.301376 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.301344 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp"] Oct 31 15:59:16.301989 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.301973 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" Oct 31 15:59:16.302052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.301991 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" Oct 31 15:59:16.302052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.302005 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" Oct 31 15:59:16.302052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.302010 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" Oct 31 15:59:16.302052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.302030 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" Oct 31 15:59:16.302052 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.302035 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" Oct 31 15:59:16.302217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.302143 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" Oct 31 15:59:16.302217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.302153 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00f75e4-410a-42c9-8908-32f009e19525" containerName="istio-proxy" Oct 31 15:59:16.302217 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.302161 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b670827b-dd4e-4e9f-938b-194b28436904" containerName="istio-proxy" Oct 31 15:59:16.306963 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.306943 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.361747 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.361717 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp"] Oct 31 15:59:16.375329 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.375301 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vbm\" (UniqueName: \"kubernetes.io/projected/112f74aa-ba8c-4799-92a7-3a06e792d4bf-kube-api-access-68vbm\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.375556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.375349 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-home\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.375556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.375376 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tmp-dir\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.375556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.375522 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-model-cache\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.375556 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.375554 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tls-certs\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.375826 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.375755 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-dshm\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.384317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.384285 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:16.408254 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.408210 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"rate-test-model-531c3c-kserve-self-signed-certs\"" Oct 31 15:59:16.408377 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.408259 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"kube-root-ca.crt\"" Oct 31 15:59:16.410784 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.410763 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"openshift-service-ca.crt\"" Oct 31 15:59:16.411076 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.410881 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-531c3c\"/\"default-dockercfg-shqgp\"" Oct 31 15:59:16.476684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.476648 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vbm\" (UniqueName: \"kubernetes.io/projected/112f74aa-ba8c-4799-92a7-3a06e792d4bf-kube-api-access-68vbm\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.476720 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-home\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.476753 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tmp-dir\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.476801 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-model-cache\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.476846 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tls-certs\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.476995 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-dshm\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.477199 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-home\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.477442 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-model-cache\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.478655 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.477592 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tmp-dir\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.481214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.481183 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tls-certs\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.481957 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.481925 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-dshm\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.533919 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.533895 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vbm\" (UniqueName: \"kubernetes.io/projected/112f74aa-ba8c-4799-92a7-3a06e792d4bf-kube-api-access-68vbm\") pod \"rate-test-model-531c3c-kserve-6bf985d8c7-rldcp\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:16.620197 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:16.620121 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:17.005624 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:17.003159 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp"] Oct 31 15:59:17.242054 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:17.242016 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" event={"ID":"112f74aa-ba8c-4799-92a7-3a06e792d4bf","Type":"ContainerStarted","Data":"070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c"} Oct 31 15:59:17.242262 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:17.242069 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" event={"ID":"112f74aa-ba8c-4799-92a7-3a06e792d4bf","Type":"ContainerStarted","Data":"8f330c968cfc7d2e5741b175e931e3abc6db333c85f611d01ac8dd7ae8b3ecbe"} Oct 31 15:59:17.358522 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:17.358408 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" podStartSLOduration=1.358393551 podStartE2EDuration="1.358393551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:59:17.356841978 +0000 UTC m=+4402.976393389" watchObservedRunningTime="2026-10-31 15:59:17.358393551 +0000 UTC m=+4402.977944961" Oct 31 15:59:17.502969 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:59:17.502934 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:59:17.621214 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:17.621091 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:18.812613 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:18.812555 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" podUID="aa9aa055-5783-4f2f-b696-7531f88f4f6e" containerName="payload-processing" containerID="cri-o://3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e" gracePeriod=25 Oct 31 15:59:18.831435 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:18.831392 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:18.921569 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:18.921511 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" podUID="7900648a-b48a-4869-a233-017d634d6e9d" containerName="payload-pre-processing" containerID="cri-o://7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6" gracePeriod=25 Oct 31 15:59:19.104683 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.104653 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:59:19.198104 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.198074 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:59:19.221085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.221037 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9aa055-5783-4f2f-b696-7531f88f4f6e-kube-api-access-rdqvb\" (UniqueName: \"kubernetes.io/projected/aa9aa055-5783-4f2f-b696-7531f88f4f6e-kube-api-access-rdqvb\") pod \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " Oct 31 15:59:19.221267 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.221252 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9aa055-5783-4f2f-b696-7531f88f4f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9aa055-5783-4f2f-b696-7531f88f4f6e-plugins-config-volume\") pod \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\" (UID: \"aa9aa055-5783-4f2f-b696-7531f88f4f6e\") " Oct 31 15:59:19.221798 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.221773 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9aa055-5783-4f2f-b696-7531f88f4f6e-plugins-config-volume" pod "aa9aa055-5783-4f2f-b696-7531f88f4f6e" (UID: "aa9aa055-5783-4f2f-b696-7531f88f4f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:59:19.224045 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.224022 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9aa055-5783-4f2f-b696-7531f88f4f6e-kube-api-access-rdqvb" pod "aa9aa055-5783-4f2f-b696-7531f88f4f6e" (UID: "aa9aa055-5783-4f2f-b696-7531f88f4f6e"). InnerVolumeSpecName "kube-api-access-rdqvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:19.258546 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.258516 4865 generic.go:356] "Generic (PLEG): container finished" podID="aa9aa055-5783-4f2f-b696-7531f88f4f6e" containerID="3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e" exitCode=0 Oct 31 15:59:19.258731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.258570 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" Oct 31 15:59:19.258731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.258603 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" event={"ID":"aa9aa055-5783-4f2f-b696-7531f88f4f6e","Type":"ContainerDied","Data":"3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e"} Oct 31 15:59:19.258731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.258673 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4" event={"ID":"aa9aa055-5783-4f2f-b696-7531f88f4f6e","Type":"ContainerDied","Data":"e982e8f97a5124a5b609a765f3cace7698dedf9a4ebfee7f54d1620c1cda0275"} Oct 31 15:59:19.258731 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.258692 4865 scope.go:122] "RemoveContainer" containerID="3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e" Oct 31 15:59:19.260544 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.260517 4865 generic.go:356] "Generic (PLEG): container finished" podID="7900648a-b48a-4869-a233-017d634d6e9d" containerID="7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6" exitCode=0 Oct 31 15:59:19.261591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.261550 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" Oct 31 15:59:19.261591 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.261558 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" event={"ID":"7900648a-b48a-4869-a233-017d634d6e9d","Type":"ContainerDied","Data":"7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6"} Oct 31 15:59:19.261758 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.261615 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl" event={"ID":"7900648a-b48a-4869-a233-017d634d6e9d","Type":"ContainerDied","Data":"df36810c42f17310df5a846d5e62260173364a1308c5c919336ca0cbe9d91462"} Oct 31 15:59:19.269902 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.269884 4865 scope.go:122] "RemoveContainer" containerID="3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e" Oct 31 15:59:19.270166 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:59:19.270148 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e\": container with ID starting with 3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e not found: ID does not exist" containerID="3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e" Oct 31 15:59:19.270222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.270175 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e"} err="failed to get container status \"3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e\": rpc error: code = NotFound desc = could not find container \"3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e\": container with ID starting with 3220233f87e37146e66bcd66be30bfbe43543cbd152b42b581df2f7d3d07ae6e not found: ID does not exist" Oct 31 15:59:19.270222 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.270192 4865 scope.go:122] "RemoveContainer" containerID="7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6" Oct 31 15:59:19.280081 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.280062 4865 scope.go:122] "RemoveContainer" containerID="7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6" Oct 31 15:59:19.280334 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:59:19.280317 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6\": container with ID starting with 7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6 not found: ID does not exist" containerID="7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6" Oct 31 15:59:19.280379 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.280341 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6"} err="failed to get container status \"7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6\": rpc error: code = NotFound desc = could not find container \"7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6\": container with ID starting with 7196d5b160a55fec93e88d3bf32768163657e6f00b170ff189d1516465e8ace6 not found: ID does not exist" Oct 31 15:59:19.323085 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.323025 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7900648a-b48a-4869-a233-017d634d6e9d-kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/7900648a-b48a-4869-a233-017d634d6e9d-kube-api-access-75qt7\") pod \"7900648a-b48a-4869-a233-017d634d6e9d\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " Oct 31 15:59:19.323207 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.323191 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7900648a-b48a-4869-a233-017d634d6e9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900648a-b48a-4869-a233-017d634d6e9d-plugins-config-volume\") pod \"7900648a-b48a-4869-a233-017d634d6e9d\" (UID: \"7900648a-b48a-4869-a233-017d634d6e9d\") " Oct 31 15:59:19.323560 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.323540 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7900648a-b48a-4869-a233-017d634d6e9d-plugins-config-volume" pod "7900648a-b48a-4869-a233-017d634d6e9d" (UID: "7900648a-b48a-4869-a233-017d634d6e9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:59:19.323751 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.323733 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9aa055-5783-4f2f-b696-7531f88f4f6e-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:19.323804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.323758 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7900648a-b48a-4869-a233-017d634d6e9d-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:19.323804 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.323772 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdqvb\" (UniqueName: \"kubernetes.io/projected/aa9aa055-5783-4f2f-b696-7531f88f4f6e-kube-api-access-rdqvb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:19.325196 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.325175 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7900648a-b48a-4869-a233-017d634d6e9d-kube-api-access-75qt7" pod "7900648a-b48a-4869-a233-017d634d6e9d" (UID: "7900648a-b48a-4869-a233-017d634d6e9d"). InnerVolumeSpecName "kube-api-access-75qt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:19.425314 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.425282 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/7900648a-b48a-4869-a233-017d634d6e9d-kube-api-access-75qt7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:19.500290 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.500261 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4"] Oct 31 15:59:19.503153 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.503130 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-24ca6e-99f5f97cf-mc5f4"] Oct 31 15:59:19.684457 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.684371 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl"] Oct 31 15:59:19.687263 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:19.687234 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-24ca6e-6887d6976c-pdqsl"] Oct 31 15:59:20.189910 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:20.189874 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:20.496970 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:20.496941 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7900648a-b48a-4869-a233-017d634d6e9d" path="/var/lib/kubelet/pods/7900648a-b48a-4869-a233-017d634d6e9d/volumes" Oct 31 15:59:20.497332 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:20.497319 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9aa055-5783-4f2f-b696-7531f88f4f6e" path="/var/lib/kubelet/pods/aa9aa055-5783-4f2f-b696-7531f88f4f6e/volumes" Oct 31 15:59:24.247696 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:24.247664 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:25.993549 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:25.993487 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:26.620704 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:26.620671 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:26.633242 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:26.633204 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:26.712830 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:26.712802 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 15:59:27.340730 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:27.340698 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 15:59:30.797257 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:30.797200 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:33.646883 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:33.646850 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:33.779141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:33.777251 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:35.192306 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:35.192270 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:35.661905 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:35.661834 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:35.800212 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:35.800179 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:36.418319 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:36.418282 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:40.697046 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:40.697007 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:41.525308 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:41.525272 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6"] Oct 31 15:59:41.634227 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:41.634183 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj"] Oct 31 15:59:42.045832 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:42.045797 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:42.166057 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:42.166023 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:42.752743 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:42.752710 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:45.662182 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:45.662147 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:46.525995 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:46.525963 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" podUID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" containerName="payload-processing" containerID="cri-o://e1d913ea8d192ab658cfff55f3387921bb8a236dc50a302804bdc3cbbb1a49ac" gracePeriod=25 Oct 31 15:59:46.634684 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:46.634643 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" podUID="feb2281f-b896-4ecb-bb3b-876a10f47803" containerName="payload-pre-processing" containerID="cri-o://98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5" gracePeriod=25 Oct 31 15:59:47.490174 ip-10-0-131-60 kubenswrapper[4865]: E1031 15:59:47.490136 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 15:59:47.535985 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:47.535951 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" podUID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:59:47.536143 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:47.536014 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" podUID="feb2281f-b896-4ecb-bb3b-876a10f47803" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:59:48.285173 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:48.285143 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:50.565505 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:50.565432 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:50.736602 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:50.736565 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:50.935838 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:50.935746 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:55.393317 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:55.393286 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:56.352141 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:56.352105 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:56.472360 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:56.472331 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 15:59:57.535350 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:57.535314 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" podUID="feb2281f-b896-4ecb-bb3b-876a10f47803" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:59:57.535783 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:57.535328 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" podUID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:59:59.806162 ip-10-0-131-60 kubenswrapper[4865]: I1031 15:59:59.806131 4865 scope.go:122] "RemoveContainer" containerID="01443565ba77f82b2a877d3b81e59f009a24d7c360f45a40e6d4d4e348d83959" Oct 31 16:00:00.098266 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.098180 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:00.178660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.178622 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6"] Oct 31 16:00:00.179574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.179547 4865 cpu_manager.go:414] "containerMap: removing container" podUID="aa9aa055-5783-4f2f-b696-7531f88f4f6e" containerName="payload-processing" Oct 31 16:00:00.179574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.179574 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9aa055-5783-4f2f-b696-7531f88f4f6e" containerName="payload-processing" Oct 31 16:00:00.179754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.179601 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7900648a-b48a-4869-a233-017d634d6e9d" containerName="payload-pre-processing" Oct 31 16:00:00.179754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.179610 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7900648a-b48a-4869-a233-017d634d6e9d" containerName="payload-pre-processing" Oct 31 16:00:00.179827 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.179761 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7900648a-b48a-4869-a233-017d634d6e9d" containerName="payload-pre-processing" Oct 31 16:00:00.179827 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.179786 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9aa055-5783-4f2f-b696-7531f88f4f6e" containerName="payload-processing" Oct 31 16:00:00.185703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.185682 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss"] Oct 31 16:00:00.185813 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.185792 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.189578 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.189554 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2"] Oct 31 16:00:00.189723 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.189672 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.193725 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.193440 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.217048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.217025 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8"] Oct 31 16:00:00.220817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.220796 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz"] Oct 31 16:00:00.220938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.220852 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.232666 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.232640 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee72658-9a5d-4896-bdcc-bc603cddd91e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.232852 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.232832 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544b4b5c-7984-4069-8022-07efaa8cd903-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.233007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.232982 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ckd\" (UniqueName: \"kubernetes.io/projected/eee72658-9a5d-4896-bdcc-bc603cddd91e-kube-api-access-p8ckd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.233130 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.233034 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjk6\" (UniqueName: \"kubernetes.io/projected/544b4b5c-7984-4069-8022-07efaa8cd903-kube-api-access-nwjk6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.233206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.233156 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jjs\" (UniqueName: \"kubernetes.io/projected/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-kube-api-access-c2jjs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.233206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.233195 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e8998-b6e0-4df4-abba-04593b1fe6bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.233334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.233243 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.233334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.233319 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j42k\" (UniqueName: \"kubernetes.io/projected/db4e8998-b6e0-4df4-abba-04593b1fe6bf-kube-api-access-7j42k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.233746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.233722 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk"] Oct 31 16:00:00.236867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.236846 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.239856 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.239837 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq"] Oct 31 16:00:00.239969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.239949 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.244004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.243979 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4"] Oct 31 16:00:00.244091 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.244038 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.248204 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.248186 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.265072 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.265050 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:00:00.265220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.265114 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:00:00.265220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.265150 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:00:00.269301 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.269281 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6"] Oct 31 16:00:00.275296 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.275273 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss"] Oct 31 16:00:00.309576 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.308958 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2"] Oct 31 16:00:00.313643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.313521 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz"] Oct 31 16:00:00.318321 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.318290 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk"] Oct 31 16:00:00.326435 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.326412 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq"] Oct 31 16:00:00.334899 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.334873 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93617e37-46b6-4692-bb88-914453c592c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.335038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.334908 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j42k\" (UniqueName: \"kubernetes.io/projected/db4e8998-b6e0-4df4-abba-04593b1fe6bf-kube-api-access-7j42k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.335038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.334947 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee72658-9a5d-4896-bdcc-bc603cddd91e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.335038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335010 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544b4b5c-7984-4069-8022-07efaa8cd903-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.335246 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335055 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e6421f-5e92-452c-9087-cb8c7f56d282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.335246 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335125 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ckd\" (UniqueName: \"kubernetes.io/projected/eee72658-9a5d-4896-bdcc-bc603cddd91e-kube-api-access-p8ckd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.335246 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335166 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjk6\" (UniqueName: \"kubernetes.io/projected/544b4b5c-7984-4069-8022-07efaa8cd903-kube-api-access-nwjk6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.335536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335282 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f89074-2a97-40ce-a080-bd14b4f8f0dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.335536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335314 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jln6s\" (UniqueName: \"kubernetes.io/projected/23e6421f-5e92-452c-9087-cb8c7f56d282-kube-api-access-jln6s\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.335536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335359 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jjs\" (UniqueName: \"kubernetes.io/projected/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-kube-api-access-c2jjs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.335536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335384 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-785nl\" (UniqueName: \"kubernetes.io/projected/93617e37-46b6-4692-bb88-914453c592c5-kube-api-access-785nl\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.335842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335707 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a48cf-44c0-4c71-96ab-600f12abe975-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.335842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335747 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee72658-9a5d-4896-bdcc-bc603cddd91e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.335842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335772 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e8998-b6e0-4df4-abba-04593b1fe6bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.335842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335806 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544b4b5c-7984-4069-8022-07efaa8cd903-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.335842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335832 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.336145 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335872 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49fd\" (UniqueName: \"kubernetes.io/projected/23f89074-2a97-40ce-a080-bd14b4f8f0dd-kube-api-access-v49fd\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.336145 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.335913 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwj5w\" (UniqueName: \"kubernetes.io/projected/645a48cf-44c0-4c71-96ab-600f12abe975-kube-api-access-kwj5w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.336315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.336296 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e8998-b6e0-4df4-abba-04593b1fe6bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.336506 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.336449 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.366002 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.365941 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4"] Oct 31 16:00:00.372643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.372619 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8"] Oct 31 16:00:00.393995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.393970 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjk6\" (UniqueName: \"kubernetes.io/projected/544b4b5c-7984-4069-8022-07efaa8cd903-kube-api-access-nwjk6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.394257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.394223 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j42k\" (UniqueName: \"kubernetes.io/projected/db4e8998-b6e0-4df4-abba-04593b1fe6bf-kube-api-access-7j42k\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.394578 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.394555 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jjs\" (UniqueName: \"kubernetes.io/projected/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-kube-api-access-c2jjs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.395062 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.395038 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ckd\" (UniqueName: \"kubernetes.io/projected/eee72658-9a5d-4896-bdcc-bc603cddd91e-kube-api-access-p8ckd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.436712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.436685 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e6421f-5e92-452c-9087-cb8c7f56d282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.436863 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.436811 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f89074-2a97-40ce-a080-bd14b4f8f0dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.436863 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.436831 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jln6s\" (UniqueName: \"kubernetes.io/projected/23e6421f-5e92-452c-9087-cb8c7f56d282-kube-api-access-jln6s\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.436987 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.436862 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-785nl\" (UniqueName: \"kubernetes.io/projected/93617e37-46b6-4692-bb88-914453c592c5-kube-api-access-785nl\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.437052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437002 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a48cf-44c0-4c71-96ab-600f12abe975-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.437114 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437092 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49fd\" (UniqueName: \"kubernetes.io/projected/23f89074-2a97-40ce-a080-bd14b4f8f0dd-kube-api-access-v49fd\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.437160 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437139 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwj5w\" (UniqueName: \"kubernetes.io/projected/645a48cf-44c0-4c71-96ab-600f12abe975-kube-api-access-kwj5w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.437209 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437192 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93617e37-46b6-4692-bb88-914453c592c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.437453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437430 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e6421f-5e92-452c-9087-cb8c7f56d282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.437685 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437661 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f89074-2a97-40ce-a080-bd14b4f8f0dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.437748 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437712 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a48cf-44c0-4c71-96ab-600f12abe975-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.437855 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.437838 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93617e37-46b6-4692-bb88-914453c592c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.494451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.494424 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-785nl\" (UniqueName: \"kubernetes.io/projected/93617e37-46b6-4692-bb88-914453c592c5-kube-api-access-785nl\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.496004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.495977 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwj5w\" (UniqueName: \"kubernetes.io/projected/645a48cf-44c0-4c71-96ab-600f12abe975-kube-api-access-kwj5w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.496004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.495996 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49fd\" (UniqueName: \"kubernetes.io/projected/23f89074-2a97-40ce-a080-bd14b4f8f0dd-kube-api-access-v49fd\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.496330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.496306 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jln6s\" (UniqueName: \"kubernetes.io/projected/23e6421f-5e92-452c-9087-cb8c7f56d282-kube-api-access-jln6s\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.501003 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.500982 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:00.509356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.509337 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:00.517223 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.517203 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:00.537741 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.537701 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:00.552683 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.552652 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:00.563387 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.563176 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:00.572513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.571290 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:00.580079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.579922 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:00.916658 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:00.916628 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544b4b5c_7984_4069_8022_07efaa8cd903.slice/crio-1998d605d9836a9c697b91c27a15bcb7e77dadfe37cc66fee76fcc070ea86104 WatchSource:0}: Error finding container 1998d605d9836a9c697b91c27a15bcb7e77dadfe37cc66fee76fcc070ea86104: Status 404 returned error can't find the container with id 1998d605d9836a9c697b91c27a15bcb7e77dadfe37cc66fee76fcc070ea86104 Oct 31 16:00:00.917141 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:00.917117 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss"] Oct 31 16:00:01.017908 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:01.017877 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f89074_2a97_40ce_a080_bd14b4f8f0dd.slice/crio-69bf92a2ba1309bde370f7f9480ef636f0c393e144a7a250b95c57883e7be692 WatchSource:0}: Error finding container 69bf92a2ba1309bde370f7f9480ef636f0c393e144a7a250b95c57883e7be692: Status 404 returned error can't find the container with id 69bf92a2ba1309bde370f7f9480ef636f0c393e144a7a250b95c57883e7be692 Oct 31 16:00:01.018924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.018895 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz"] Oct 31 16:00:01.081175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.081009 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4"] Oct 31 16:00:01.081633 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:01.081591 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93617e37_46b6_4692_bb88_914453c592c5.slice/crio-078c0343049788e9647e437729ec0d1dc8593fc60d2c0e0f4a8538edfe6ac46f WatchSource:0}: Error finding container 078c0343049788e9647e437729ec0d1dc8593fc60d2c0e0f4a8538edfe6ac46f: Status 404 returned error can't find the container with id 078c0343049788e9647e437729ec0d1dc8593fc60d2c0e0f4a8538edfe6ac46f Oct 31 16:00:01.083463 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:01.083442 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645a48cf_44c0_4c71_96ab_600f12abe975.slice/crio-5d8be14d2216e6bf2ba9b95646e7d264297797fb3a1bf57990b7016a54b83198 WatchSource:0}: Error finding container 5d8be14d2216e6bf2ba9b95646e7d264297797fb3a1bf57990b7016a54b83198: Status 404 returned error can't find the container with id 5d8be14d2216e6bf2ba9b95646e7d264297797fb3a1bf57990b7016a54b83198 Oct 31 16:00:01.084970 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:01.084941 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e6421f_5e92_452c_9087_cb8c7f56d282.slice/crio-48aa73d7834300925c4dfa12f0740a8d20bcb7c740fa21f4bd9b012294cc1248 WatchSource:0}: Error finding container 48aa73d7834300925c4dfa12f0740a8d20bcb7c740fa21f4bd9b012294cc1248: Status 404 returned error can't find the container with id 48aa73d7834300925c4dfa12f0740a8d20bcb7c740fa21f4bd9b012294cc1248 Oct 31 16:00:01.085777 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.085757 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk"] Oct 31 16:00:01.089547 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.089526 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq"] Oct 31 16:00:01.125836 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.125809 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6"] Oct 31 16:00:01.126162 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:01.126136 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4e8998_b6e0_4df4_abba_04593b1fe6bf.slice/crio-5abe58ddd589fe708db09dafdf3f328f618d27f73263d933c07225ebe447dc88 WatchSource:0}: Error finding container 5abe58ddd589fe708db09dafdf3f328f618d27f73263d933c07225ebe447dc88: Status 404 returned error can't find the container with id 5abe58ddd589fe708db09dafdf3f328f618d27f73263d933c07225ebe447dc88 Oct 31 16:00:01.140330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.140309 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2"] Oct 31 16:00:01.140911 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:01.140879 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee72658_9a5d_4896_bdcc_bc603cddd91e.slice/crio-3320d99cf9105df1770f0c444c2889a2712b30fe5d7f10d5529c5af0448bda63 WatchSource:0}: Error finding container 3320d99cf9105df1770f0c444c2889a2712b30fe5d7f10d5529c5af0448bda63: Status 404 returned error can't find the container with id 3320d99cf9105df1770f0c444c2889a2712b30fe5d7f10d5529c5af0448bda63 Oct 31 16:00:01.146999 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.146977 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8"] Oct 31 16:00:01.161766 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:01.161741 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c35ac9b_c9da_453b_950b_cd9a97b12dcf.slice/crio-55dd704d4bd8eed0c90eecd008b4c933552b3439340f6a16cef957a5ca4fe9e2 WatchSource:0}: Error finding container 55dd704d4bd8eed0c90eecd008b4c933552b3439340f6a16cef957a5ca4fe9e2: Status 404 returned error can't find the container with id 55dd704d4bd8eed0c90eecd008b4c933552b3439340f6a16cef957a5ca4fe9e2 Oct 31 16:00:01.550840 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.550809 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" event={"ID":"0c35ac9b-c9da-453b-950b-cd9a97b12dcf","Type":"ContainerStarted","Data":"c06fe1601cc5168078455dd6c1a890e6b2e6018e0c0daa47e94c72d1d9cbe415"} Oct 31 16:00:01.551046 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.550894 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" event={"ID":"0c35ac9b-c9da-453b-950b-cd9a97b12dcf","Type":"ContainerStarted","Data":"55dd704d4bd8eed0c90eecd008b4c933552b3439340f6a16cef957a5ca4fe9e2"} Oct 31 16:00:01.552551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.552523 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" event={"ID":"23f89074-2a97-40ce-a080-bd14b4f8f0dd","Type":"ContainerStarted","Data":"103f70e01dd166ac2081937cd5b72e36f660c865c0438039084b4c618013d5c5"} Oct 31 16:00:01.552695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.552566 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" event={"ID":"23f89074-2a97-40ce-a080-bd14b4f8f0dd","Type":"ContainerStarted","Data":"69bf92a2ba1309bde370f7f9480ef636f0c393e144a7a250b95c57883e7be692"} Oct 31 16:00:01.553806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.553783 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" event={"ID":"93617e37-46b6-4692-bb88-914453c592c5","Type":"ContainerStarted","Data":"078c0343049788e9647e437729ec0d1dc8593fc60d2c0e0f4a8538edfe6ac46f"} Oct 31 16:00:01.555428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.555389 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" event={"ID":"544b4b5c-7984-4069-8022-07efaa8cd903","Type":"ContainerStarted","Data":"e4ab2a9d1e21bbb7e91178cd696d550dd5815a42567566561635f04ce2cd2a3d"} Oct 31 16:00:01.555579 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.555513 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" event={"ID":"544b4b5c-7984-4069-8022-07efaa8cd903","Type":"ContainerStarted","Data":"1998d605d9836a9c697b91c27a15bcb7e77dadfe37cc66fee76fcc070ea86104"} Oct 31 16:00:01.556824 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.556801 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" event={"ID":"645a48cf-44c0-4c71-96ab-600f12abe975","Type":"ContainerStarted","Data":"5d8be14d2216e6bf2ba9b95646e7d264297797fb3a1bf57990b7016a54b83198"} Oct 31 16:00:01.557919 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.557900 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" event={"ID":"23e6421f-5e92-452c-9087-cb8c7f56d282","Type":"ContainerStarted","Data":"48aa73d7834300925c4dfa12f0740a8d20bcb7c740fa21f4bd9b012294cc1248"} Oct 31 16:00:01.559095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.559075 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" event={"ID":"eee72658-9a5d-4896-bdcc-bc603cddd91e","Type":"ContainerStarted","Data":"3320d99cf9105df1770f0c444c2889a2712b30fe5d7f10d5529c5af0448bda63"} Oct 31 16:00:01.560216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.560197 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" event={"ID":"db4e8998-b6e0-4df4-abba-04593b1fe6bf","Type":"ContainerStarted","Data":"5abe58ddd589fe708db09dafdf3f328f618d27f73263d933c07225ebe447dc88"} Oct 31 16:00:01.668014 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.667940 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" podStartSLOduration=1.667925844 podStartE2EDuration="1.667925844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:01.664520204 +0000 UTC m=+4447.284071618" watchObservedRunningTime="2026-10-31 16:00:01.667925844 +0000 UTC m=+4447.287477258" Oct 31 16:00:01.781788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.781728 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" podStartSLOduration=1.781715738 podStartE2EDuration="1.781715738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:01.779017778 +0000 UTC m=+4447.398569189" watchObservedRunningTime="2026-10-31 16:00:01.781715738 +0000 UTC m=+4447.401267149" Oct 31 16:00:01.898293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:01.898211 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" podStartSLOduration=1.898187612 podStartE2EDuration="1.898187612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:01.894823678 +0000 UTC m=+4447.514375091" watchObservedRunningTime="2026-10-31 16:00:01.898187612 +0000 UTC m=+4447.517739024" Oct 31 16:00:02.567937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.567902 4865 generic.go:356] "Generic (PLEG): container finished" podID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerID="103f70e01dd166ac2081937cd5b72e36f660c865c0438039084b4c618013d5c5" exitCode=60 Oct 31 16:00:02.568424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.567985 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" event={"ID":"23f89074-2a97-40ce-a080-bd14b4f8f0dd","Type":"ContainerDied","Data":"103f70e01dd166ac2081937cd5b72e36f660c865c0438039084b4c618013d5c5"} Oct 31 16:00:02.568424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.568221 4865 scope.go:122] "RemoveContainer" containerID="103f70e01dd166ac2081937cd5b72e36f660c865c0438039084b4c618013d5c5" Oct 31 16:00:02.569750 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.569730 4865 generic.go:356] "Generic (PLEG): container finished" podID="93617e37-46b6-4692-bb88-914453c592c5" containerID="038a55efb94b162ae652b662c0afc32c091be23cc46dd05b06f70d6f42becbd6" exitCode=60 Oct 31 16:00:02.569861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.569834 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" event={"ID":"93617e37-46b6-4692-bb88-914453c592c5","Type":"ContainerDied","Data":"038a55efb94b162ae652b662c0afc32c091be23cc46dd05b06f70d6f42becbd6"} Oct 31 16:00:02.570148 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.570127 4865 scope.go:122] "RemoveContainer" containerID="038a55efb94b162ae652b662c0afc32c091be23cc46dd05b06f70d6f42becbd6" Oct 31 16:00:02.571653 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.571486 4865 generic.go:356] "Generic (PLEG): container finished" podID="544b4b5c-7984-4069-8022-07efaa8cd903" containerID="e4ab2a9d1e21bbb7e91178cd696d550dd5815a42567566561635f04ce2cd2a3d" exitCode=60 Oct 31 16:00:02.571653 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.571564 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" event={"ID":"544b4b5c-7984-4069-8022-07efaa8cd903","Type":"ContainerDied","Data":"e4ab2a9d1e21bbb7e91178cd696d550dd5815a42567566561635f04ce2cd2a3d"} Oct 31 16:00:02.572020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.572000 4865 scope.go:122] "RemoveContainer" containerID="e4ab2a9d1e21bbb7e91178cd696d550dd5815a42567566561635f04ce2cd2a3d" Oct 31 16:00:02.573342 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.573252 4865 generic.go:356] "Generic (PLEG): container finished" podID="645a48cf-44c0-4c71-96ab-600f12abe975" containerID="00aab9315a09df75f3580eb4a19493ce9ba6e5c7688cfe1fde59ef6443abcc75" exitCode=60 Oct 31 16:00:02.573342 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.573274 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" event={"ID":"645a48cf-44c0-4c71-96ab-600f12abe975","Type":"ContainerDied","Data":"00aab9315a09df75f3580eb4a19493ce9ba6e5c7688cfe1fde59ef6443abcc75"} Oct 31 16:00:02.573545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.573524 4865 scope.go:122] "RemoveContainer" containerID="00aab9315a09df75f3580eb4a19493ce9ba6e5c7688cfe1fde59ef6443abcc75" Oct 31 16:00:02.574927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.574903 4865 generic.go:356] "Generic (PLEG): container finished" podID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerID="6e930693d6da8ce50fc221dfbcb5b37c879fa6bfd02dbb75dd32e05d87074ac2" exitCode=60 Oct 31 16:00:02.575031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.574972 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" event={"ID":"23e6421f-5e92-452c-9087-cb8c7f56d282","Type":"ContainerDied","Data":"6e930693d6da8ce50fc221dfbcb5b37c879fa6bfd02dbb75dd32e05d87074ac2"} Oct 31 16:00:02.575208 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.575192 4865 scope.go:122] "RemoveContainer" containerID="6e930693d6da8ce50fc221dfbcb5b37c879fa6bfd02dbb75dd32e05d87074ac2" Oct 31 16:00:02.576447 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.576428 4865 generic.go:356] "Generic (PLEG): container finished" podID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerID="6f576e21b6cd1c169ac628c8475964fc1c0302cbed38ba3da85445168c8b8a7d" exitCode=60 Oct 31 16:00:02.576551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.576499 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" event={"ID":"eee72658-9a5d-4896-bdcc-bc603cddd91e","Type":"ContainerDied","Data":"6f576e21b6cd1c169ac628c8475964fc1c0302cbed38ba3da85445168c8b8a7d"} Oct 31 16:00:02.576767 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.576751 4865 scope.go:122] "RemoveContainer" containerID="6f576e21b6cd1c169ac628c8475964fc1c0302cbed38ba3da85445168c8b8a7d" Oct 31 16:00:02.578167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.578149 4865 generic.go:356] "Generic (PLEG): container finished" podID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerID="d6d40863e16e5738058b37f7db0b531ab84ade800807b60d2b4bd3213f587ce3" exitCode=60 Oct 31 16:00:02.578271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.578255 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" event={"ID":"db4e8998-b6e0-4df4-abba-04593b1fe6bf","Type":"ContainerDied","Data":"d6d40863e16e5738058b37f7db0b531ab84ade800807b60d2b4bd3213f587ce3"} Oct 31 16:00:02.578607 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.578548 4865 scope.go:122] "RemoveContainer" containerID="d6d40863e16e5738058b37f7db0b531ab84ade800807b60d2b4bd3213f587ce3" Oct 31 16:00:02.581451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.581429 4865 generic.go:356] "Generic (PLEG): container finished" podID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerID="c06fe1601cc5168078455dd6c1a890e6b2e6018e0c0daa47e94c72d1d9cbe415" exitCode=60 Oct 31 16:00:02.581577 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.581526 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" event={"ID":"0c35ac9b-c9da-453b-950b-cd9a97b12dcf","Type":"ContainerDied","Data":"c06fe1601cc5168078455dd6c1a890e6b2e6018e0c0daa47e94c72d1d9cbe415"} Oct 31 16:00:02.581760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:02.581745 4865 scope.go:122] "RemoveContainer" containerID="c06fe1601cc5168078455dd6c1a890e6b2e6018e0c0daa47e94c72d1d9cbe415" Oct 31 16:00:03.590526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.590488 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" event={"ID":"544b4b5c-7984-4069-8022-07efaa8cd903","Type":"ContainerStarted","Data":"f96db10f68f9c76da76ff91395f6f49b2290471cdb9cc4c35c8bd11945af9b8b"} Oct 31 16:00:03.592398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.592374 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" event={"ID":"645a48cf-44c0-4c71-96ab-600f12abe975","Type":"ContainerStarted","Data":"6e25c503263fd002330186c4742182b9384e0099d1d6efdd599e5d15ac35bda2"} Oct 31 16:00:03.594822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.594753 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" event={"ID":"23e6421f-5e92-452c-9087-cb8c7f56d282","Type":"ContainerStarted","Data":"3a1793333e8ea01d4a99d51462f3a6e96d9a6205647d1c289b62e5439a44e9d9"} Oct 31 16:00:03.603081 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.603057 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" event={"ID":"eee72658-9a5d-4896-bdcc-bc603cddd91e","Type":"ContainerStarted","Data":"9b4befd0bbe765c232937fa9bfc29297fe721c40be39a7c1d3adcee123b9c3f1"} Oct 31 16:00:03.605973 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.605951 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" event={"ID":"db4e8998-b6e0-4df4-abba-04593b1fe6bf","Type":"ContainerStarted","Data":"e221eb32e46e2274fa7f4ef1526e9ea4ed3ac0bd81500852a5eaf1bc0b733784"} Oct 31 16:00:03.608004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.607982 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" event={"ID":"0c35ac9b-c9da-453b-950b-cd9a97b12dcf","Type":"ContainerStarted","Data":"fc207625f8b6f7770e7566a289e3778be60843872f128dde7f0c9c11fb685592"} Oct 31 16:00:03.609990 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.609969 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" event={"ID":"23f89074-2a97-40ce-a080-bd14b4f8f0dd","Type":"ContainerStarted","Data":"45a91745ceb7963e643415a059eb2866fafad4b7b4bf5cc5679f68986ba3b672"} Oct 31 16:00:03.612103 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.612081 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" event={"ID":"93617e37-46b6-4692-bb88-914453c592c5","Type":"ContainerStarted","Data":"bbd5bd62161b0f99578a56afc69f9f4462c31f1a9682ac9c422f55702d23fb5e"} Oct 31 16:00:03.823890 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.823847 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" podStartSLOduration=3.823832566 podStartE2EDuration="3.823832566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:03.821381679 +0000 UTC m=+4449.440933091" watchObservedRunningTime="2026-10-31 16:00:03.823832566 +0000 UTC m=+4449.443383976" Oct 31 16:00:03.937635 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:03.937588 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" podStartSLOduration=3.937573868 podStartE2EDuration="3.937573868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:03.935894703 +0000 UTC m=+4449.555446115" watchObservedRunningTime="2026-10-31 16:00:03.937573868 +0000 UTC m=+4449.557125313" Oct 31 16:00:04.053693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.053643 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" podStartSLOduration=4.053626326 podStartE2EDuration="4.053626326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:04.050928295 +0000 UTC m=+4449.670479706" watchObservedRunningTime="2026-10-31 16:00:04.053626326 +0000 UTC m=+4449.673177737" Oct 31 16:00:04.285048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.284994 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" podStartSLOduration=4.2849807030000004 podStartE2EDuration="4.284980703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:04.281811968 +0000 UTC m=+4449.901363382" watchObservedRunningTime="2026-10-31 16:00:04.284980703 +0000 UTC m=+4449.904532114" Oct 31 16:00:04.512157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.512104 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" podStartSLOduration=4.512089638 podStartE2EDuration="4.512089638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:04.508860229 +0000 UTC m=+4450.128411641" watchObservedRunningTime="2026-10-31 16:00:04.512089638 +0000 UTC m=+4450.131641049" Oct 31 16:00:04.621517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.621427 4865 generic.go:356] "Generic (PLEG): container finished" podID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerID="9b4befd0bbe765c232937fa9bfc29297fe721c40be39a7c1d3adcee123b9c3f1" exitCode=60 Oct 31 16:00:04.621517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.621503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" event={"ID":"eee72658-9a5d-4896-bdcc-bc603cddd91e","Type":"ContainerDied","Data":"9b4befd0bbe765c232937fa9bfc29297fe721c40be39a7c1d3adcee123b9c3f1"} Oct 31 16:00:04.622021 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.621574 4865 scope.go:122] "RemoveContainer" containerID="6f576e21b6cd1c169ac628c8475964fc1c0302cbed38ba3da85445168c8b8a7d" Oct 31 16:00:04.622021 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.621756 4865 scope.go:122] "RemoveContainer" containerID="9b4befd0bbe765c232937fa9bfc29297fe721c40be39a7c1d3adcee123b9c3f1" Oct 31 16:00:04.622110 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.622065 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2_odh-ai-gateway-infra(eee72658-9a5d-4896-bdcc-bc603cddd91e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" Oct 31 16:00:04.623713 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.623685 4865 generic.go:356] "Generic (PLEG): container finished" podID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerID="e221eb32e46e2274fa7f4ef1526e9ea4ed3ac0bd81500852a5eaf1bc0b733784" exitCode=60 Oct 31 16:00:04.623835 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.623768 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" event={"ID":"db4e8998-b6e0-4df4-abba-04593b1fe6bf","Type":"ContainerDied","Data":"e221eb32e46e2274fa7f4ef1526e9ea4ed3ac0bd81500852a5eaf1bc0b733784"} Oct 31 16:00:04.623980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.623965 4865 scope.go:122] "RemoveContainer" containerID="e221eb32e46e2274fa7f4ef1526e9ea4ed3ac0bd81500852a5eaf1bc0b733784" Oct 31 16:00:04.624286 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.624254 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6_odh-ai-gateway-infra(db4e8998-b6e0-4df4-abba-04593b1fe6bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" Oct 31 16:00:04.625526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.625503 4865 generic.go:356] "Generic (PLEG): container finished" podID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerID="fc207625f8b6f7770e7566a289e3778be60843872f128dde7f0c9c11fb685592" exitCode=60 Oct 31 16:00:04.625612 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.625580 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" event={"ID":"0c35ac9b-c9da-453b-950b-cd9a97b12dcf","Type":"ContainerDied","Data":"fc207625f8b6f7770e7566a289e3778be60843872f128dde7f0c9c11fb685592"} Oct 31 16:00:04.625790 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.625769 4865 scope.go:122] "RemoveContainer" containerID="fc207625f8b6f7770e7566a289e3778be60843872f128dde7f0c9c11fb685592" Oct 31 16:00:04.626055 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.626034 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8_odh-ai-gateway-infra(0c35ac9b-c9da-453b-950b-cd9a97b12dcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" Oct 31 16:00:04.627518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.627498 4865 generic.go:356] "Generic (PLEG): container finished" podID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerID="45a91745ceb7963e643415a059eb2866fafad4b7b4bf5cc5679f68986ba3b672" exitCode=60 Oct 31 16:00:04.627611 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.627569 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" event={"ID":"23f89074-2a97-40ce-a080-bd14b4f8f0dd","Type":"ContainerDied","Data":"45a91745ceb7963e643415a059eb2866fafad4b7b4bf5cc5679f68986ba3b672"} Oct 31 16:00:04.627760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.627743 4865 scope.go:122] "RemoveContainer" containerID="45a91745ceb7963e643415a059eb2866fafad4b7b4bf5cc5679f68986ba3b672" Oct 31 16:00:04.628009 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.627986 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz_odh-ai-gateway-infra(23f89074-2a97-40ce-a080-bd14b4f8f0dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" Oct 31 16:00:04.629721 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.629700 4865 generic.go:356] "Generic (PLEG): container finished" podID="93617e37-46b6-4692-bb88-914453c592c5" containerID="bbd5bd62161b0f99578a56afc69f9f4462c31f1a9682ac9c422f55702d23fb5e" exitCode=60 Oct 31 16:00:04.629843 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.629757 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" event={"ID":"93617e37-46b6-4692-bb88-914453c592c5","Type":"ContainerDied","Data":"bbd5bd62161b0f99578a56afc69f9f4462c31f1a9682ac9c422f55702d23fb5e"} Oct 31 16:00:04.629984 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.629966 4865 scope.go:122] "RemoveContainer" containerID="bbd5bd62161b0f99578a56afc69f9f4462c31f1a9682ac9c422f55702d23fb5e" Oct 31 16:00:04.630227 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.630207 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4_odh-ai-gateway-infra(93617e37-46b6-4692-bb88-914453c592c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" podUID="93617e37-46b6-4692-bb88-914453c592c5" Oct 31 16:00:04.632180 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.632155 4865 generic.go:356] "Generic (PLEG): container finished" podID="544b4b5c-7984-4069-8022-07efaa8cd903" containerID="f96db10f68f9c76da76ff91395f6f49b2290471cdb9cc4c35c8bd11945af9b8b" exitCode=60 Oct 31 16:00:04.632504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.632457 4865 scope.go:122] "RemoveContainer" containerID="f96db10f68f9c76da76ff91395f6f49b2290471cdb9cc4c35c8bd11945af9b8b" Oct 31 16:00:04.632590 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.632252 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" event={"ID":"544b4b5c-7984-4069-8022-07efaa8cd903","Type":"ContainerDied","Data":"f96db10f68f9c76da76ff91395f6f49b2290471cdb9cc4c35c8bd11945af9b8b"} Oct 31 16:00:04.632750 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.632724 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss_odh-ai-gateway-infra(544b4b5c-7984-4069-8022-07efaa8cd903)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" Oct 31 16:00:04.634578 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.634556 4865 generic.go:356] "Generic (PLEG): container finished" podID="645a48cf-44c0-4c71-96ab-600f12abe975" containerID="6e25c503263fd002330186c4742182b9384e0099d1d6efdd599e5d15ac35bda2" exitCode=60 Oct 31 16:00:04.634694 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.634639 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" event={"ID":"645a48cf-44c0-4c71-96ab-600f12abe975","Type":"ContainerDied","Data":"6e25c503263fd002330186c4742182b9384e0099d1d6efdd599e5d15ac35bda2"} Oct 31 16:00:04.634934 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.634913 4865 scope.go:122] "RemoveContainer" containerID="6e25c503263fd002330186c4742182b9384e0099d1d6efdd599e5d15ac35bda2" Oct 31 16:00:04.635183 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.635151 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq_odh-ai-gateway-infra(645a48cf-44c0-4c71-96ab-600f12abe975)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" Oct 31 16:00:04.636701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.636668 4865 generic.go:356] "Generic (PLEG): container finished" podID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerID="3a1793333e8ea01d4a99d51462f3a6e96d9a6205647d1c289b62e5439a44e9d9" exitCode=60 Oct 31 16:00:04.636796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.636712 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" event={"ID":"23e6421f-5e92-452c-9087-cb8c7f56d282","Type":"ContainerDied","Data":"3a1793333e8ea01d4a99d51462f3a6e96d9a6205647d1c289b62e5439a44e9d9"} Oct 31 16:00:04.637033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.636999 4865 scope.go:122] "RemoveContainer" containerID="3a1793333e8ea01d4a99d51462f3a6e96d9a6205647d1c289b62e5439a44e9d9" Oct 31 16:00:04.637124 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.637037 4865 scope.go:122] "RemoveContainer" containerID="d6d40863e16e5738058b37f7db0b531ab84ade800807b60d2b4bd3213f587ce3" Oct 31 16:00:04.637223 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:04.637198 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk_odh-ai-gateway-infra(23e6421f-5e92-452c-9087-cb8c7f56d282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" Oct 31 16:00:04.650933 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.650911 4865 scope.go:122] "RemoveContainer" containerID="c06fe1601cc5168078455dd6c1a890e6b2e6018e0c0daa47e94c72d1d9cbe415" Oct 31 16:00:04.663540 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.663523 4865 scope.go:122] "RemoveContainer" containerID="103f70e01dd166ac2081937cd5b72e36f660c865c0438039084b4c618013d5c5" Oct 31 16:00:04.677618 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.677568 4865 scope.go:122] "RemoveContainer" containerID="038a55efb94b162ae652b662c0afc32c091be23cc46dd05b06f70d6f42becbd6" Oct 31 16:00:04.698064 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.698040 4865 scope.go:122] "RemoveContainer" containerID="e4ab2a9d1e21bbb7e91178cd696d550dd5815a42567566561635f04ce2cd2a3d" Oct 31 16:00:04.711531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.711512 4865 scope.go:122] "RemoveContainer" containerID="00aab9315a09df75f3580eb4a19493ce9ba6e5c7688cfe1fde59ef6443abcc75" Oct 31 16:00:04.724516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.724461 4865 scope.go:122] "RemoveContainer" containerID="6e930693d6da8ce50fc221dfbcb5b37c879fa6bfd02dbb75dd32e05d87074ac2" Oct 31 16:00:04.902662 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:04.902588 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:05.649431 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.649350 4865 scope.go:122] "RemoveContainer" containerID="3a1793333e8ea01d4a99d51462f3a6e96d9a6205647d1c289b62e5439a44e9d9" Oct 31 16:00:05.649932 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.649686 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk_odh-ai-gateway-infra(23e6421f-5e92-452c-9087-cb8c7f56d282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" Oct 31 16:00:05.651293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.651269 4865 scope.go:122] "RemoveContainer" containerID="9b4befd0bbe765c232937fa9bfc29297fe721c40be39a7c1d3adcee123b9c3f1" Oct 31 16:00:05.651607 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.651581 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2_odh-ai-gateway-infra(eee72658-9a5d-4896-bdcc-bc603cddd91e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" Oct 31 16:00:05.652989 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.652967 4865 scope.go:122] "RemoveContainer" containerID="e221eb32e46e2274fa7f4ef1526e9ea4ed3ac0bd81500852a5eaf1bc0b733784" Oct 31 16:00:05.653231 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.653212 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6_odh-ai-gateway-infra(db4e8998-b6e0-4df4-abba-04593b1fe6bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" Oct 31 16:00:05.654514 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.654498 4865 scope.go:122] "RemoveContainer" containerID="fc207625f8b6f7770e7566a289e3778be60843872f128dde7f0c9c11fb685592" Oct 31 16:00:05.654731 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.654708 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8_odh-ai-gateway-infra(0c35ac9b-c9da-453b-950b-cd9a97b12dcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" Oct 31 16:00:05.656065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.656043 4865 scope.go:122] "RemoveContainer" containerID="45a91745ceb7963e643415a059eb2866fafad4b7b4bf5cc5679f68986ba3b672" Oct 31 16:00:05.656289 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.656269 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz_odh-ai-gateway-infra(23f89074-2a97-40ce-a080-bd14b4f8f0dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" Oct 31 16:00:05.657598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.657579 4865 scope.go:122] "RemoveContainer" containerID="bbd5bd62161b0f99578a56afc69f9f4462c31f1a9682ac9c422f55702d23fb5e" Oct 31 16:00:05.657787 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.657766 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4_odh-ai-gateway-infra(93617e37-46b6-4692-bb88-914453c592c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" podUID="93617e37-46b6-4692-bb88-914453c592c5" Oct 31 16:00:05.659138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.659119 4865 scope.go:122] "RemoveContainer" containerID="f96db10f68f9c76da76ff91395f6f49b2290471cdb9cc4c35c8bd11945af9b8b" Oct 31 16:00:05.659323 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.659300 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss_odh-ai-gateway-infra(544b4b5c-7984-4069-8022-07efaa8cd903)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" Oct 31 16:00:05.660616 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:05.660598 4865 scope.go:122] "RemoveContainer" containerID="6e25c503263fd002330186c4742182b9384e0099d1d6efdd599e5d15ac35bda2" Oct 31 16:00:05.660800 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:05.660781 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq_odh-ai-gateway-infra(645a48cf-44c0-4c71-96ab-600f12abe975)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" Oct 31 16:00:06.427736 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.427700 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:00:06.671301 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.671270 4865 generic.go:356] "Generic (PLEG): container finished" podID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" containerID="e1d913ea8d192ab658cfff55f3387921bb8a236dc50a302804bdc3cbbb1a49ac" exitCode=0 Oct 31 16:00:06.671924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.671494 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" event={"ID":"28a82de8-9d25-4b34-8edc-b03ab1e1b689","Type":"ContainerDied","Data":"e1d913ea8d192ab658cfff55f3387921bb8a236dc50a302804bdc3cbbb1a49ac"} Oct 31 16:00:06.810052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.810018 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 16:00:06.910306 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.910278 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 16:00:06.960260 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.960215 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a82de8-9d25-4b34-8edc-b03ab1e1b689-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a82de8-9d25-4b34-8edc-b03ab1e1b689-plugins-config-volume\") pod \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " Oct 31 16:00:06.960494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.960283 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a82de8-9d25-4b34-8edc-b03ab1e1b689-kube-api-access-p5gn4\" (UniqueName: \"kubernetes.io/projected/28a82de8-9d25-4b34-8edc-b03ab1e1b689-kube-api-access-p5gn4\") pod \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\" (UID: \"28a82de8-9d25-4b34-8edc-b03ab1e1b689\") " Oct 31 16:00:06.961735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.961703 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a82de8-9d25-4b34-8edc-b03ab1e1b689-plugins-config-volume" pod "28a82de8-9d25-4b34-8edc-b03ab1e1b689" (UID: "28a82de8-9d25-4b34-8edc-b03ab1e1b689"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:06.991517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:06.990695 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a82de8-9d25-4b34-8edc-b03ab1e1b689-kube-api-access-p5gn4" pod "28a82de8-9d25-4b34-8edc-b03ab1e1b689" (UID: "28a82de8-9d25-4b34-8edc-b03ab1e1b689"). InnerVolumeSpecName "kube-api-access-p5gn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:07.062195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.062163 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb2281f-b896-4ecb-bb3b-876a10f47803-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb2281f-b896-4ecb-bb3b-876a10f47803-plugins-config-volume\") pod \"feb2281f-b896-4ecb-bb3b-876a10f47803\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " Oct 31 16:00:07.062411 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.062393 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb2281f-b896-4ecb-bb3b-876a10f47803-kube-api-access-qskww\" (UniqueName: \"kubernetes.io/projected/feb2281f-b896-4ecb-bb3b-876a10f47803-kube-api-access-qskww\") pod \"feb2281f-b896-4ecb-bb3b-876a10f47803\" (UID: \"feb2281f-b896-4ecb-bb3b-876a10f47803\") " Oct 31 16:00:07.062587 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.062565 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb2281f-b896-4ecb-bb3b-876a10f47803-plugins-config-volume" pod "feb2281f-b896-4ecb-bb3b-876a10f47803" (UID: "feb2281f-b896-4ecb-bb3b-876a10f47803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:07.062858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.062838 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb2281f-b896-4ecb-bb3b-876a10f47803-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:07.062858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.062860 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a82de8-9d25-4b34-8edc-b03ab1e1b689-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:07.063066 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.062875 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gn4\" (UniqueName: \"kubernetes.io/projected/28a82de8-9d25-4b34-8edc-b03ab1e1b689-kube-api-access-p5gn4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:07.065444 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.065416 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb2281f-b896-4ecb-bb3b-876a10f47803-kube-api-access-qskww" pod "feb2281f-b896-4ecb-bb3b-876a10f47803" (UID: "feb2281f-b896-4ecb-bb3b-876a10f47803"). InnerVolumeSpecName "kube-api-access-qskww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:07.163897 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.163867 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qskww\" (UniqueName: \"kubernetes.io/projected/feb2281f-b896-4ecb-bb3b-876a10f47803-kube-api-access-qskww\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:07.681033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.680999 4865 generic.go:356] "Generic (PLEG): container finished" podID="feb2281f-b896-4ecb-bb3b-876a10f47803" containerID="98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5" exitCode=0 Oct 31 16:00:07.681625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.681069 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" Oct 31 16:00:07.681625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.681085 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" event={"ID":"feb2281f-b896-4ecb-bb3b-876a10f47803","Type":"ContainerDied","Data":"98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5"} Oct 31 16:00:07.681625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.681149 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj" event={"ID":"feb2281f-b896-4ecb-bb3b-876a10f47803","Type":"ContainerDied","Data":"a872e6306082fad040c61fac3011fd8ca410bafbce9237ee4fe3cd074df1f7a5"} Oct 31 16:00:07.681625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.681177 4865 scope.go:122] "RemoveContainer" containerID="98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5" Oct 31 16:00:07.682859 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.682841 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" Oct 31 16:00:07.682973 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.682860 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6" event={"ID":"28a82de8-9d25-4b34-8edc-b03ab1e1b689","Type":"ContainerDied","Data":"7e2d7e4fd316d96a4ba9a0c2ead8a199b5d27f25daeed3e13de1e78c7bdc2a82"} Oct 31 16:00:07.695330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.695306 4865 scope.go:122] "RemoveContainer" containerID="98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5" Oct 31 16:00:07.695724 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:07.695704 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5\": container with ID starting with 98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5 not found: ID does not exist" containerID="98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5" Oct 31 16:00:07.695802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.695729 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5"} err="failed to get container status \"98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5\": rpc error: code = NotFound desc = could not find container \"98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5\": container with ID starting with 98a67307895ad14ee905cce11a19494e987ee1868777dea632411cf6a9735cd5 not found: ID does not exist" Oct 31 16:00:07.695802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.695747 4865 scope.go:122] "RemoveContainer" containerID="e1d913ea8d192ab658cfff55f3387921bb8a236dc50a302804bdc3cbbb1a49ac" Oct 31 16:00:07.848180 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.848091 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6"] Oct 31 16:00:07.851203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.851183 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c947d9-975c7f8c-kq9w6"] Oct 31 16:00:07.960950 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.960916 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj"] Oct 31 16:00:07.967070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:07.967034 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c947d9-5dffc575f-jhbwj"] Oct 31 16:00:08.497560 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:08.497529 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" path="/var/lib/kubelet/pods/28a82de8-9d25-4b34-8edc-b03ab1e1b689/volumes" Oct 31 16:00:08.497936 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:08.497922 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb2281f-b896-4ecb-bb3b-876a10f47803" path="/var/lib/kubelet/pods/feb2281f-b896-4ecb-bb3b-876a10f47803/volumes" Oct 31 16:00:09.625275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:09.625247 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:10.385873 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:10.385838 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:11.897498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:11.897434 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:14.326287 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:14.326251 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:14.493183 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:14.493156 4865 scope.go:122] "RemoveContainer" containerID="e221eb32e46e2274fa7f4ef1526e9ea4ed3ac0bd81500852a5eaf1bc0b733784" Oct 31 16:00:15.489090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:15.489060 4865 scope.go:122] "RemoveContainer" containerID="6e25c503263fd002330186c4742182b9384e0099d1d6efdd599e5d15ac35bda2" Oct 31 16:00:15.489625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:15.489237 4865 scope.go:122] "RemoveContainer" containerID="45a91745ceb7963e643415a059eb2866fafad4b7b4bf5cc5679f68986ba3b672" Oct 31 16:00:15.747721 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:15.747689 4865 generic.go:356] "Generic (PLEG): container finished" podID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerID="410563254a20987f1dd117119eba3738b11cbaa033644c3b9e6ad5c0b20835e8" exitCode=60 Oct 31 16:00:15.747870 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:15.747757 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" event={"ID":"db4e8998-b6e0-4df4-abba-04593b1fe6bf","Type":"ContainerDied","Data":"410563254a20987f1dd117119eba3738b11cbaa033644c3b9e6ad5c0b20835e8"} Oct 31 16:00:15.747870 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:15.747818 4865 scope.go:122] "RemoveContainer" containerID="e221eb32e46e2274fa7f4ef1526e9ea4ed3ac0bd81500852a5eaf1bc0b733784" Oct 31 16:00:15.748282 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:15.748251 4865 scope.go:122] "RemoveContainer" containerID="410563254a20987f1dd117119eba3738b11cbaa033644c3b9e6ad5c0b20835e8" Oct 31 16:00:15.748666 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:15.748643 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6_odh-ai-gateway-infra(db4e8998-b6e0-4df4-abba-04593b1fe6bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" Oct 31 16:00:16.498375 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.498341 4865 scope.go:122] "RemoveContainer" containerID="fc207625f8b6f7770e7566a289e3778be60843872f128dde7f0c9c11fb685592" Oct 31 16:00:16.499258 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.498527 4865 scope.go:122] "RemoveContainer" containerID="9b4befd0bbe765c232937fa9bfc29297fe721c40be39a7c1d3adcee123b9c3f1" Oct 31 16:00:16.758683 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.758610 4865 generic.go:356] "Generic (PLEG): container finished" podID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerID="bce701b8b11fabe181f817dd9c0da80dd98366264a53934f4aafbe78b1a27ca7" exitCode=60 Oct 31 16:00:16.758836 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.758681 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" event={"ID":"23f89074-2a97-40ce-a080-bd14b4f8f0dd","Type":"ContainerDied","Data":"bce701b8b11fabe181f817dd9c0da80dd98366264a53934f4aafbe78b1a27ca7"} Oct 31 16:00:16.758836 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.758745 4865 scope.go:122] "RemoveContainer" containerID="45a91745ceb7963e643415a059eb2866fafad4b7b4bf5cc5679f68986ba3b672" Oct 31 16:00:16.759183 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.759146 4865 scope.go:122] "RemoveContainer" containerID="bce701b8b11fabe181f817dd9c0da80dd98366264a53934f4aafbe78b1a27ca7" Oct 31 16:00:16.759576 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:16.759550 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz_odh-ai-gateway-infra(23f89074-2a97-40ce-a080-bd14b4f8f0dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" Oct 31 16:00:16.761285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.761259 4865 generic.go:356] "Generic (PLEG): container finished" podID="645a48cf-44c0-4c71-96ab-600f12abe975" containerID="94bdb0b8c3579502aec8273557d2337c90d121d574453aae3ec8297f005f0a72" exitCode=60 Oct 31 16:00:16.761398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.761376 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" event={"ID":"645a48cf-44c0-4c71-96ab-600f12abe975","Type":"ContainerDied","Data":"94bdb0b8c3579502aec8273557d2337c90d121d574453aae3ec8297f005f0a72"} Oct 31 16:00:16.761770 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.761754 4865 scope.go:122] "RemoveContainer" containerID="94bdb0b8c3579502aec8273557d2337c90d121d574453aae3ec8297f005f0a72" Oct 31 16:00:16.762063 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:16.762038 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq_odh-ai-gateway-infra(645a48cf-44c0-4c71-96ab-600f12abe975)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" Oct 31 16:00:16.774606 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.774591 4865 scope.go:122] "RemoveContainer" containerID="6e25c503263fd002330186c4742182b9384e0099d1d6efdd599e5d15ac35bda2" Oct 31 16:00:16.928451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:16.928421 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6"] Oct 31 16:00:17.090011 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.089983 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:17.190114 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.190083 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4e8998-b6e0-4df4-abba-04593b1fe6bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e8998-b6e0-4df4-abba-04593b1fe6bf-maas-api-service-ca\") pod \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " Oct 31 16:00:17.190292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.190181 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4e8998-b6e0-4df4-abba-04593b1fe6bf-kube-api-access-7j42k\" (UniqueName: \"kubernetes.io/projected/db4e8998-b6e0-4df4-abba-04593b1fe6bf-kube-api-access-7j42k\") pod \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\" (UID: \"db4e8998-b6e0-4df4-abba-04593b1fe6bf\") " Oct 31 16:00:17.190521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.190497 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4e8998-b6e0-4df4-abba-04593b1fe6bf-maas-api-service-ca" pod "db4e8998-b6e0-4df4-abba-04593b1fe6bf" (UID: "db4e8998-b6e0-4df4-abba-04593b1fe6bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:17.190743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.190729 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4e8998-b6e0-4df4-abba-04593b1fe6bf-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:17.192590 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.192562 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4e8998-b6e0-4df4-abba-04593b1fe6bf-kube-api-access-7j42k" pod "db4e8998-b6e0-4df4-abba-04593b1fe6bf" (UID: "db4e8998-b6e0-4df4-abba-04593b1fe6bf"). InnerVolumeSpecName "kube-api-access-7j42k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:17.291704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.291674 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j42k\" (UniqueName: \"kubernetes.io/projected/db4e8998-b6e0-4df4-abba-04593b1fe6bf-kube-api-access-7j42k\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:17.490021 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.489986 4865 scope.go:122] "RemoveContainer" containerID="3a1793333e8ea01d4a99d51462f3a6e96d9a6205647d1c289b62e5439a44e9d9" Oct 31 16:00:17.490244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.490209 4865 scope.go:122] "RemoveContainer" containerID="f96db10f68f9c76da76ff91395f6f49b2290471cdb9cc4c35c8bd11945af9b8b" Oct 31 16:00:17.490384 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.490344 4865 scope.go:122] "RemoveContainer" containerID="bbd5bd62161b0f99578a56afc69f9f4462c31f1a9682ac9c422f55702d23fb5e" Oct 31 16:00:17.491682 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:17.491654 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:00:17.774825 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.774799 4865 generic.go:356] "Generic (PLEG): container finished" podID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerID="db21f575c3f74adf9651e8a3994cf94584f4790a1709ee91597a2acf50a4ff0f" exitCode=60 Oct 31 16:00:17.775204 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.774872 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" event={"ID":"eee72658-9a5d-4896-bdcc-bc603cddd91e","Type":"ContainerDied","Data":"db21f575c3f74adf9651e8a3994cf94584f4790a1709ee91597a2acf50a4ff0f"} Oct 31 16:00:17.775204 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.774929 4865 scope.go:122] "RemoveContainer" containerID="9b4befd0bbe765c232937fa9bfc29297fe721c40be39a7c1d3adcee123b9c3f1" Oct 31 16:00:17.775347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.775329 4865 scope.go:122] "RemoveContainer" containerID="db21f575c3f74adf9651e8a3994cf94584f4790a1709ee91597a2acf50a4ff0f" Oct 31 16:00:17.776152 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:17.776107 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2_odh-ai-gateway-infra(eee72658-9a5d-4896-bdcc-bc603cddd91e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" Oct 31 16:00:17.777715 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.777688 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" event={"ID":"db4e8998-b6e0-4df4-abba-04593b1fe6bf","Type":"ContainerDied","Data":"5abe58ddd589fe708db09dafdf3f328f618d27f73263d933c07225ebe447dc88"} Oct 31 16:00:17.777862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.777847 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6" Oct 31 16:00:17.789793 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.789766 4865 generic.go:356] "Generic (PLEG): container finished" podID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerID="eada879573f23b7feccd0b86b9e68626606988098f55dc7bdcb10e00c1a19a5d" exitCode=60 Oct 31 16:00:17.789903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.789815 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" event={"ID":"0c35ac9b-c9da-453b-950b-cd9a97b12dcf","Type":"ContainerDied","Data":"eada879573f23b7feccd0b86b9e68626606988098f55dc7bdcb10e00c1a19a5d"} Oct 31 16:00:17.790191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.790176 4865 scope.go:122] "RemoveContainer" containerID="eada879573f23b7feccd0b86b9e68626606988098f55dc7bdcb10e00c1a19a5d" Oct 31 16:00:17.790509 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:17.790487 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8_odh-ai-gateway-infra(0c35ac9b-c9da-453b-950b-cd9a97b12dcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" Oct 31 16:00:17.795106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.795086 4865 scope.go:122] "RemoveContainer" containerID="410563254a20987f1dd117119eba3738b11cbaa033644c3b9e6ad5c0b20835e8" Oct 31 16:00:17.804669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.804652 4865 scope.go:122] "RemoveContainer" containerID="fc207625f8b6f7770e7566a289e3778be60843872f128dde7f0c9c11fb685592" Oct 31 16:00:17.885097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:17.885076 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz"] Oct 31 16:00:18.003903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.003876 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq"] Oct 31 16:00:18.049153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.049126 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:18.103094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.103058 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f89074-2a97-40ce-a080-bd14b4f8f0dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f89074-2a97-40ce-a080-bd14b4f8f0dd-maas-api-service-ca\") pod \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " Oct 31 16:00:18.103248 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.103232 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f89074-2a97-40ce-a080-bd14b4f8f0dd-kube-api-access-v49fd\" (UniqueName: \"kubernetes.io/projected/23f89074-2a97-40ce-a080-bd14b4f8f0dd-kube-api-access-v49fd\") pod \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\" (UID: \"23f89074-2a97-40ce-a080-bd14b4f8f0dd\") " Oct 31 16:00:18.104831 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.104796 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f89074-2a97-40ce-a080-bd14b4f8f0dd-maas-api-service-ca" pod "23f89074-2a97-40ce-a080-bd14b4f8f0dd" (UID: "23f89074-2a97-40ce-a080-bd14b4f8f0dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:18.106398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.106369 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f89074-2a97-40ce-a080-bd14b4f8f0dd-kube-api-access-v49fd" pod "23f89074-2a97-40ce-a080-bd14b4f8f0dd" (UID: "23f89074-2a97-40ce-a080-bd14b4f8f0dd"). InnerVolumeSpecName "kube-api-access-v49fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:18.154079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.154055 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:18.206097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.206066 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645a48cf-44c0-4c71-96ab-600f12abe975-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a48cf-44c0-4c71-96ab-600f12abe975-maas-api-service-ca\") pod \"645a48cf-44c0-4c71-96ab-600f12abe975\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " Oct 31 16:00:18.206265 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.206124 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645a48cf-44c0-4c71-96ab-600f12abe975-kube-api-access-kwj5w\" (UniqueName: \"kubernetes.io/projected/645a48cf-44c0-4c71-96ab-600f12abe975-kube-api-access-kwj5w\") pod \"645a48cf-44c0-4c71-96ab-600f12abe975\" (UID: \"645a48cf-44c0-4c71-96ab-600f12abe975\") " Oct 31 16:00:18.206488 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.206443 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645a48cf-44c0-4c71-96ab-600f12abe975-maas-api-service-ca" pod "645a48cf-44c0-4c71-96ab-600f12abe975" (UID: "645a48cf-44c0-4c71-96ab-600f12abe975"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:18.206715 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.206694 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645a48cf-44c0-4c71-96ab-600f12abe975-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:18.206788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.206720 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v49fd\" (UniqueName: \"kubernetes.io/projected/23f89074-2a97-40ce-a080-bd14b4f8f0dd-kube-api-access-v49fd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:18.206788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.206736 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23f89074-2a97-40ce-a080-bd14b4f8f0dd-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:18.208294 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.208275 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645a48cf-44c0-4c71-96ab-600f12abe975-kube-api-access-kwj5w" pod "645a48cf-44c0-4c71-96ab-600f12abe975" (UID: "645a48cf-44c0-4c71-96ab-600f12abe975"). InnerVolumeSpecName "kube-api-access-kwj5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:18.240421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.240391 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6"] Oct 31 16:00:18.243723 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.243695 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891040-czqj6"] Oct 31 16:00:18.308162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.308082 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwj5w\" (UniqueName: \"kubernetes.io/projected/645a48cf-44c0-4c71-96ab-600f12abe975-kube-api-access-kwj5w\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:18.501676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.501641 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" path="/var/lib/kubelet/pods/db4e8998-b6e0-4df4-abba-04593b1fe6bf/volumes" Oct 31 16:00:18.797241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.797201 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" event={"ID":"23f89074-2a97-40ce-a080-bd14b4f8f0dd","Type":"ContainerDied","Data":"69bf92a2ba1309bde370f7f9480ef636f0c393e144a7a250b95c57883e7be692"} Oct 31 16:00:18.797769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.797276 4865 scope.go:122] "RemoveContainer" containerID="bce701b8b11fabe181f817dd9c0da80dd98366264a53934f4aafbe78b1a27ca7" Oct 31 16:00:18.797769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.797207 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz" Oct 31 16:00:18.799703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.799679 4865 generic.go:356] "Generic (PLEG): container finished" podID="93617e37-46b6-4692-bb88-914453c592c5" containerID="81bfe95dfc8d7aaa2ec7a02db68a90407ef299c1ff167fddc0c3b87a69309b4a" exitCode=60 Oct 31 16:00:18.799844 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.799785 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" event={"ID":"93617e37-46b6-4692-bb88-914453c592c5","Type":"ContainerDied","Data":"81bfe95dfc8d7aaa2ec7a02db68a90407ef299c1ff167fddc0c3b87a69309b4a"} Oct 31 16:00:18.800313 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.800291 4865 scope.go:122] "RemoveContainer" containerID="81bfe95dfc8d7aaa2ec7a02db68a90407ef299c1ff167fddc0c3b87a69309b4a" Oct 31 16:00:18.800667 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:18.800633 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4_odh-ai-gateway-infra(93617e37-46b6-4692-bb88-914453c592c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" podUID="93617e37-46b6-4692-bb88-914453c592c5" Oct 31 16:00:18.802204 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.802182 4865 generic.go:356] "Generic (PLEG): container finished" podID="544b4b5c-7984-4069-8022-07efaa8cd903" containerID="8fa022db26c214c533d2046b556206888d03451a978586fd76e4cc1733fbc61d" exitCode=60 Oct 31 16:00:18.802308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.802263 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" event={"ID":"544b4b5c-7984-4069-8022-07efaa8cd903","Type":"ContainerDied","Data":"8fa022db26c214c533d2046b556206888d03451a978586fd76e4cc1733fbc61d"} Oct 31 16:00:18.802682 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.802667 4865 scope.go:122] "RemoveContainer" containerID="8fa022db26c214c533d2046b556206888d03451a978586fd76e4cc1733fbc61d" Oct 31 16:00:18.802954 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:18.802931 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss_odh-ai-gateway-infra(544b4b5c-7984-4069-8022-07efaa8cd903)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" Oct 31 16:00:18.803679 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.803661 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" Oct 31 16:00:18.803768 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.803706 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq" event={"ID":"645a48cf-44c0-4c71-96ab-600f12abe975","Type":"ContainerDied","Data":"5d8be14d2216e6bf2ba9b95646e7d264297797fb3a1bf57990b7016a54b83198"} Oct 31 16:00:18.806134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.806115 4865 generic.go:356] "Generic (PLEG): container finished" podID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerID="d9e0971f0110202e40ca503883103efad788532e5323acfd55347e9393225fcf" exitCode=60 Oct 31 16:00:18.806247 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.806191 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" event={"ID":"23e6421f-5e92-452c-9087-cb8c7f56d282","Type":"ContainerDied","Data":"d9e0971f0110202e40ca503883103efad788532e5323acfd55347e9393225fcf"} Oct 31 16:00:18.806608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.806591 4865 scope.go:122] "RemoveContainer" containerID="d9e0971f0110202e40ca503883103efad788532e5323acfd55347e9393225fcf" Oct 31 16:00:18.806873 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:18.806852 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk_odh-ai-gateway-infra(23e6421f-5e92-452c-9087-cb8c7f56d282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" Oct 31 16:00:18.809075 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.809060 4865 scope.go:122] "RemoveContainer" containerID="bbd5bd62161b0f99578a56afc69f9f4462c31f1a9682ac9c422f55702d23fb5e" Oct 31 16:00:18.822538 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.822516 4865 scope.go:122] "RemoveContainer" containerID="f96db10f68f9c76da76ff91395f6f49b2290471cdb9cc4c35c8bd11945af9b8b" Oct 31 16:00:18.835179 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.835150 4865 scope.go:122] "RemoveContainer" containerID="94bdb0b8c3579502aec8273557d2337c90d121d574453aae3ec8297f005f0a72" Oct 31 16:00:18.844156 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.844139 4865 scope.go:122] "RemoveContainer" containerID="3a1793333e8ea01d4a99d51462f3a6e96d9a6205647d1c289b62e5439a44e9d9" Oct 31 16:00:18.907362 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:18.907329 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2"] Oct 31 16:00:19.049699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.049665 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:19.059972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.059952 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:19.085866 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.085794 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz"] Oct 31 16:00:19.091031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.091009 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891040-78tqz"] Oct 31 16:00:19.118561 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.118524 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee72658-9a5d-4896-bdcc-bc603cddd91e-kube-api-access-p8ckd\" (UniqueName: \"kubernetes.io/projected/eee72658-9a5d-4896-bdcc-bc603cddd91e-kube-api-access-p8ckd\") pod \"eee72658-9a5d-4896-bdcc-bc603cddd91e\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " Oct 31 16:00:19.118735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.118621 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee72658-9a5d-4896-bdcc-bc603cddd91e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee72658-9a5d-4896-bdcc-bc603cddd91e-maas-api-service-ca\") pod \"eee72658-9a5d-4896-bdcc-bc603cddd91e\" (UID: \"eee72658-9a5d-4896-bdcc-bc603cddd91e\") " Oct 31 16:00:19.119001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.118978 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee72658-9a5d-4896-bdcc-bc603cddd91e-maas-api-service-ca" pod "eee72658-9a5d-4896-bdcc-bc603cddd91e" (UID: "eee72658-9a5d-4896-bdcc-bc603cddd91e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:19.121191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.121166 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee72658-9a5d-4896-bdcc-bc603cddd91e-kube-api-access-p8ckd" pod "eee72658-9a5d-4896-bdcc-bc603cddd91e" (UID: "eee72658-9a5d-4896-bdcc-bc603cddd91e"). InnerVolumeSpecName "kube-api-access-p8ckd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:19.137049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.137022 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8"] Oct 31 16:00:19.220784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.220744 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ckd\" (UniqueName: \"kubernetes.io/projected/eee72658-9a5d-4896-bdcc-bc603cddd91e-kube-api-access-p8ckd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:19.220995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.220788 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee72658-9a5d-4896-bdcc-bc603cddd91e-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:19.302805 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.302781 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:19.422954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.422868 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-maas-api-service-ca\") pod \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " Oct 31 16:00:19.423140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.422955 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-kube-api-access-c2jjs\" (UniqueName: \"kubernetes.io/projected/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-kube-api-access-c2jjs\") pod \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\" (UID: \"0c35ac9b-c9da-453b-950b-cd9a97b12dcf\") " Oct 31 16:00:19.423298 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.423274 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-maas-api-service-ca" pod "0c35ac9b-c9da-453b-950b-cd9a97b12dcf" (UID: "0c35ac9b-c9da-453b-950b-cd9a97b12dcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:19.423568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.423548 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:19.425210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.425190 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-kube-api-access-c2jjs" pod "0c35ac9b-c9da-453b-950b-cd9a97b12dcf" (UID: "0c35ac9b-c9da-453b-950b-cd9a97b12dcf"). InnerVolumeSpecName "kube-api-access-c2jjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:19.432311 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.432289 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq"] Oct 31 16:00:19.435605 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.435584 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-31dff8-29891040-4zzjq"] Oct 31 16:00:19.524928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.524895 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2jjs\" (UniqueName: \"kubernetes.io/projected/0c35ac9b-c9da-453b-950b-cd9a97b12dcf-kube-api-access-c2jjs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:19.822766 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.822734 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" Oct 31 16:00:19.822766 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.822753 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2" event={"ID":"eee72658-9a5d-4896-bdcc-bc603cddd91e","Type":"ContainerDied","Data":"3320d99cf9105df1770f0c444c2889a2712b30fe5d7f10d5529c5af0448bda63"} Oct 31 16:00:19.823338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.822809 4865 scope.go:122] "RemoveContainer" containerID="db21f575c3f74adf9651e8a3994cf94584f4790a1709ee91597a2acf50a4ff0f" Oct 31 16:00:19.824161 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.824142 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" event={"ID":"0c35ac9b-c9da-453b-950b-cd9a97b12dcf","Type":"ContainerDied","Data":"55dd704d4bd8eed0c90eecd008b4c933552b3439340f6a16cef957a5ca4fe9e2"} Oct 31 16:00:19.824224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.824210 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8" Oct 31 16:00:19.833560 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.833538 4865 scope.go:122] "RemoveContainer" containerID="eada879573f23b7feccd0b86b9e68626606988098f55dc7bdcb10e00c1a19a5d" Oct 31 16:00:19.983240 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.983206 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss"] Oct 31 16:00:19.990892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.990852 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8"] Oct 31 16:00:19.998065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:19.998029 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-140c48-29891040-bdbz8"] Oct 31 16:00:20.108339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.108307 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2"] Oct 31 16:00:20.111372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.111348 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891040-6mbm2"] Oct 31 16:00:20.138485 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.138453 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:20.215335 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.215308 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk"] Oct 31 16:00:20.232891 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.232850 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544b4b5c-7984-4069-8022-07efaa8cd903-kube-api-access-nwjk6\" (UniqueName: \"kubernetes.io/projected/544b4b5c-7984-4069-8022-07efaa8cd903-kube-api-access-nwjk6\") pod \"544b4b5c-7984-4069-8022-07efaa8cd903\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " Oct 31 16:00:20.233120 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.233106 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544b4b5c-7984-4069-8022-07efaa8cd903-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544b4b5c-7984-4069-8022-07efaa8cd903-maas-api-service-ca\") pod \"544b4b5c-7984-4069-8022-07efaa8cd903\" (UID: \"544b4b5c-7984-4069-8022-07efaa8cd903\") " Oct 31 16:00:20.233482 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.233429 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544b4b5c-7984-4069-8022-07efaa8cd903-maas-api-service-ca" pod "544b4b5c-7984-4069-8022-07efaa8cd903" (UID: "544b4b5c-7984-4069-8022-07efaa8cd903"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:20.235112 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.235089 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544b4b5c-7984-4069-8022-07efaa8cd903-kube-api-access-nwjk6" pod "544b4b5c-7984-4069-8022-07efaa8cd903" (UID: "544b4b5c-7984-4069-8022-07efaa8cd903"). InnerVolumeSpecName "kube-api-access-nwjk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:20.328996 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.328966 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4"] Oct 31 16:00:20.334446 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.334416 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544b4b5c-7984-4069-8022-07efaa8cd903-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:20.334583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.334448 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwjk6\" (UniqueName: \"kubernetes.io/projected/544b4b5c-7984-4069-8022-07efaa8cd903-kube-api-access-nwjk6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:20.388419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.388382 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:20.435834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.435801 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e6421f-5e92-452c-9087-cb8c7f56d282-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e6421f-5e92-452c-9087-cb8c7f56d282-maas-api-service-ca\") pod \"23e6421f-5e92-452c-9087-cb8c7f56d282\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " Oct 31 16:00:20.436016 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.435845 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e6421f-5e92-452c-9087-cb8c7f56d282-kube-api-access-jln6s\" (UniqueName: \"kubernetes.io/projected/23e6421f-5e92-452c-9087-cb8c7f56d282-kube-api-access-jln6s\") pod \"23e6421f-5e92-452c-9087-cb8c7f56d282\" (UID: \"23e6421f-5e92-452c-9087-cb8c7f56d282\") " Oct 31 16:00:20.436212 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.436184 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e6421f-5e92-452c-9087-cb8c7f56d282-maas-api-service-ca" pod "23e6421f-5e92-452c-9087-cb8c7f56d282" (UID: "23e6421f-5e92-452c-9087-cb8c7f56d282"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:20.436376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.436359 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e6421f-5e92-452c-9087-cb8c7f56d282-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:20.438138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.438116 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e6421f-5e92-452c-9087-cb8c7f56d282-kube-api-access-jln6s" pod "23e6421f-5e92-452c-9087-cb8c7f56d282" (UID: "23e6421f-5e92-452c-9087-cb8c7f56d282"). InnerVolumeSpecName "kube-api-access-jln6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:20.493241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.493216 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:20.497078 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.497052 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" path="/var/lib/kubelet/pods/0c35ac9b-c9da-453b-950b-cd9a97b12dcf/volumes" Oct 31 16:00:20.497503 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.497488 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" path="/var/lib/kubelet/pods/23f89074-2a97-40ce-a080-bd14b4f8f0dd/volumes" Oct 31 16:00:20.497864 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.497852 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" path="/var/lib/kubelet/pods/645a48cf-44c0-4c71-96ab-600f12abe975/volumes" Oct 31 16:00:20.498529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.498514 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" path="/var/lib/kubelet/pods/eee72658-9a5d-4896-bdcc-bc603cddd91e/volumes" Oct 31 16:00:20.537211 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.537185 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93617e37-46b6-4692-bb88-914453c592c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93617e37-46b6-4692-bb88-914453c592c5-maas-api-service-ca\") pod \"93617e37-46b6-4692-bb88-914453c592c5\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " Oct 31 16:00:20.537306 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.537240 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93617e37-46b6-4692-bb88-914453c592c5-kube-api-access-785nl\" (UniqueName: \"kubernetes.io/projected/93617e37-46b6-4692-bb88-914453c592c5-kube-api-access-785nl\") pod \"93617e37-46b6-4692-bb88-914453c592c5\" (UID: \"93617e37-46b6-4692-bb88-914453c592c5\") " Oct 31 16:00:20.537620 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.537593 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93617e37-46b6-4692-bb88-914453c592c5-maas-api-service-ca" pod "93617e37-46b6-4692-bb88-914453c592c5" (UID: "93617e37-46b6-4692-bb88-914453c592c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:20.537923 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.537898 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jln6s\" (UniqueName: \"kubernetes.io/projected/23e6421f-5e92-452c-9087-cb8c7f56d282-kube-api-access-jln6s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:20.538043 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.537927 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93617e37-46b6-4692-bb88-914453c592c5-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:20.539668 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.539646 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93617e37-46b6-4692-bb88-914453c592c5-kube-api-access-785nl" pod "93617e37-46b6-4692-bb88-914453c592c5" (UID: "93617e37-46b6-4692-bb88-914453c592c5"). InnerVolumeSpecName "kube-api-access-785nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:20.639957 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.639877 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-785nl\" (UniqueName: \"kubernetes.io/projected/93617e37-46b6-4692-bb88-914453c592c5-kube-api-access-785nl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:20.840285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.840251 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" Oct 31 16:00:20.840794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.840260 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss" event={"ID":"544b4b5c-7984-4069-8022-07efaa8cd903","Type":"ContainerDied","Data":"1998d605d9836a9c697b91c27a15bcb7e77dadfe37cc66fee76fcc070ea86104"} Oct 31 16:00:20.840794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.840398 4865 scope.go:122] "RemoveContainer" containerID="8fa022db26c214c533d2046b556206888d03451a978586fd76e4cc1733fbc61d" Oct 31 16:00:20.842919 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.842902 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" Oct 31 16:00:20.843041 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.842953 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk" event={"ID":"23e6421f-5e92-452c-9087-cb8c7f56d282","Type":"ContainerDied","Data":"48aa73d7834300925c4dfa12f0740a8d20bcb7c740fa21f4bd9b012294cc1248"} Oct 31 16:00:20.846967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.846949 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" event={"ID":"93617e37-46b6-4692-bb88-914453c592c5","Type":"ContainerDied","Data":"078c0343049788e9647e437729ec0d1dc8593fc60d2c0e0f4a8538edfe6ac46f"} Oct 31 16:00:20.847058 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.846985 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4" Oct 31 16:00:20.852121 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.852104 4865 scope.go:122] "RemoveContainer" containerID="d9e0971f0110202e40ca503883103efad788532e5323acfd55347e9393225fcf" Oct 31 16:00:20.861986 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:20.861969 4865 scope.go:122] "RemoveContainer" containerID="81bfe95dfc8d7aaa2ec7a02db68a90407ef299c1ff167fddc0c3b87a69309b4a" Oct 31 16:00:21.007136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:21.007092 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss"] Oct 31 16:00:21.012562 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:21.012523 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-531c3c-29891040-59tss"] Oct 31 16:00:21.123055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:21.123024 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk"] Oct 31 16:00:21.127498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:21.127442 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29891040-tczpk"] Oct 31 16:00:21.240428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:21.240396 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4"] Oct 31 16:00:21.243155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:21.243133 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891040-fnbf4"] Oct 31 16:00:22.496664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:22.496623 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" path="/var/lib/kubelet/pods/23e6421f-5e92-452c-9087-cb8c7f56d282/volumes" Oct 31 16:00:22.497284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:22.497262 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" path="/var/lib/kubelet/pods/544b4b5c-7984-4069-8022-07efaa8cd903/volumes" Oct 31 16:00:22.497765 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:22.497745 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93617e37-46b6-4692-bb88-914453c592c5" path="/var/lib/kubelet/pods/93617e37-46b6-4692-bb88-914453c592c5/volumes" Oct 31 16:00:23.785355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:23.785321 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:27.141350 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141320 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7"] Oct 31 16:00:27.141892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141879 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.141935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141894 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.141935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141908 4865 cpu_manager.go:414] "containerMap: removing container" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:27.141935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141912 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:27.141935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141921 4865 cpu_manager.go:414] "containerMap: removing container" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.141935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141925 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.141935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141934 4865 cpu_manager.go:414] "containerMap: removing container" podUID="feb2281f-b896-4ecb-bb3b-876a10f47803" containerName="payload-pre-processing" Oct 31 16:00:27.141935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141939 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb2281f-b896-4ecb-bb3b-876a10f47803" containerName="payload-pre-processing" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141948 4865 cpu_manager.go:414] "containerMap: removing container" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141966 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141973 4865 cpu_manager.go:414] "containerMap: removing container" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141978 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141988 4865 cpu_manager.go:414] "containerMap: removing container" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.141993 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142003 4865 cpu_manager.go:414] "containerMap: removing container" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142008 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142013 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142018 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142029 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142033 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142043 4865 cpu_manager.go:414] "containerMap: removing container" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142047 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142054 4865 cpu_manager.go:414] "containerMap: removing container" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142058 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142064 4865 cpu_manager.go:414] "containerMap: removing container" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142068 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142075 4865 cpu_manager.go:414] "containerMap: removing container" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142080 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142085 4865 cpu_manager.go:414] "containerMap: removing container" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142088 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142095 4865 cpu_manager.go:414] "containerMap: removing container" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142099 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142106 4865 cpu_manager.go:414] "containerMap: removing container" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142110 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142117 4865 cpu_manager.go:414] "containerMap: removing container" podUID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" containerName="payload-processing" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142121 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" containerName="payload-processing" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142127 4865 cpu_manager.go:414] "containerMap: removing container" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142131 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142143 4865 cpu_manager.go:414] "containerMap: removing container" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142147 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142154 4865 cpu_manager.go:414] "containerMap: removing container" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142158 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142166 4865 cpu_manager.go:414] "containerMap: removing container" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:27.142194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142171 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142255 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142263 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142271 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142277 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142285 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142291 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142297 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142302 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142308 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142312 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142317 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142324 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142329 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142333 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a82de8-9d25-4b34-8edc-b03ab1e1b689" containerName="payload-processing" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142338 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142343 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142348 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142353 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142360 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142368 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb2281f-b896-4ecb-bb3b-876a10f47803" containerName="payload-pre-processing" Oct 31 16:00:27.143359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.142373 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:27.147298 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.147279 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.202819 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.202769 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7"] Oct 31 16:00:27.219319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.219289 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-dshm\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.219529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.219330 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-home\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.219529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.219352 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fb26\" (UniqueName: \"kubernetes.io/projected/ab2c6505-e58e-40b6-9f64-5236f9dd5035-kube-api-access-5fb26\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.219529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.219370 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tmp-dir\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.219741 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.219614 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tls-certs\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.219741 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.219690 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-model-cache\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.225638 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.225612 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:27.249623 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.249596 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"kube-root-ca.crt\"" Oct 31 16:00:27.249623 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.249609 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"openshift-service-ca.crt\"" Oct 31 16:00:27.249795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.249634 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"rate-test-model-55b3fb-kserve-self-signed-certs\"" Oct 31 16:00:27.251639 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.251623 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-55b3fb\"/\"default-dockercfg-8kj8z\"" Oct 31 16:00:27.321079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321049 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-dshm\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321090 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-home\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321112 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fb26\" (UniqueName: \"kubernetes.io/projected/ab2c6505-e58e-40b6-9f64-5236f9dd5035-kube-api-access-5fb26\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321132 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tmp-dir\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321210 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tls-certs\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321252 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-model-cache\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321645 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321620 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-home\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321713 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321673 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tmp-dir\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.321713 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.321689 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-model-cache\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.323889 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.323865 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-dshm\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.324038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.323905 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tls-certs\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.379293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.379260 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fb26\" (UniqueName: \"kubernetes.io/projected/ab2c6505-e58e-40b6-9f64-5236f9dd5035-kube-api-access-5fb26\") pod \"rate-test-model-55b3fb-kserve-67f86dd49-h5qf7\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.459268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.459227 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:27.839195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.839165 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7"] Oct 31 16:00:27.839529 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:27.839503 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab2c6505_e58e_40b6_9f64_5236f9dd5035.slice/crio-6cf95a87280da5f24fcc3a50d37150a6a9444b7a73adc2c3da600258a1660f79 WatchSource:0}: Error finding container 6cf95a87280da5f24fcc3a50d37150a6a9444b7a73adc2c3da600258a1660f79: Status 404 returned error can't find the container with id 6cf95a87280da5f24fcc3a50d37150a6a9444b7a73adc2c3da600258a1660f79 Oct 31 16:00:27.854091 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.854057 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:27.897861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:27.897833 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" event={"ID":"ab2c6505-e58e-40b6-9f64-5236f9dd5035","Type":"ContainerStarted","Data":"6cf95a87280da5f24fcc3a50d37150a6a9444b7a73adc2c3da600258a1660f79"} Oct 31 16:00:28.906831 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:28.906794 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" event={"ID":"ab2c6505-e58e-40b6-9f64-5236f9dd5035","Type":"ContainerStarted","Data":"833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db"} Oct 31 16:00:29.024613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:29.024556 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" podStartSLOduration=2.024541876 podStartE2EDuration="2.024541876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:29.022757792 +0000 UTC m=+4474.642309203" watchObservedRunningTime="2026-10-31 16:00:29.024541876 +0000 UTC m=+4474.644093286" Oct 31 16:00:29.459595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:29.459562 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:29.547934 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:29.547902 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:30.961527 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:30.961497 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:31.085982 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:31.085937 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:32.474769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:32.474735 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:32.933661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:32.933577 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:33.371241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:33.371194 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:34.046328 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046290 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd"] Oct 31 16:00:34.046916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046899 4865 cpu_manager.go:414] "containerMap: removing container" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:34.046991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046918 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f89074-2a97-40ce-a080-bd14b4f8f0dd" containerName="cleanup" Oct 31 16:00:34.046991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046944 4865 cpu_manager.go:414] "containerMap: removing container" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:34.046991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046949 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:34.046991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046958 4865 cpu_manager.go:414] "containerMap: removing container" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:34.046991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046963 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645a48cf-44c0-4c71-96ab-600f12abe975" containerName="cleanup" Oct 31 16:00:34.047178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.046995 4865 cpu_manager.go:414] "containerMap: removing container" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:34.047178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.047000 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544b4b5c-7984-4069-8022-07efaa8cd903" containerName="cleanup" Oct 31 16:00:34.047178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.047077 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4e8998-b6e0-4df4-abba-04593b1fe6bf" containerName="cleanup" Oct 31 16:00:34.047178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.047087 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e6421f-5e92-452c-9087-cb8c7f56d282" containerName="cleanup" Oct 31 16:00:34.047178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.047096 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="93617e37-46b6-4692-bb88-914453c592c5" containerName="cleanup" Oct 31 16:00:34.047178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.047107 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee72658-9a5d-4896-bdcc-bc603cddd91e" containerName="cleanup" Oct 31 16:00:34.047178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.047115 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c35ac9b-c9da-453b-950b-cd9a97b12dcf" containerName="cleanup" Oct 31 16:00:34.050276 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.050259 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.106075 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.106043 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd"] Oct 31 16:00:34.153293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.153255 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:00:34.153293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.153255 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:00:34.153549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.153302 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:00:34.154462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.154447 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:00:34.228219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.228179 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.228419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.228296 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.228419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.228389 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9l9\" (UniqueName: \"kubernetes.io/projected/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-kube-api-access-bd9l9\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.329998 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.329894 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.330175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.330019 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.330175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.330120 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9l9\" (UniqueName: \"kubernetes.io/projected/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-kube-api-access-bd9l9\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.330815 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.330793 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.332555 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.332532 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.388057 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.388013 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9l9\" (UniqueName: \"kubernetes.io/projected/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-kube-api-access-bd9l9\") pod \"maas-api-revoke-sub-badaa017-wnjgd\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.665031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.664938 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:34.685355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:34.685319 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:35.045838 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:35.045810 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd"] Oct 31 16:00:35.046821 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:35.046794 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c9a085_d4a7_4874_a6f7_ac9e127bbefb.slice/crio-d42678f0309ff56a12f540d7469a670aeb0eccf0c1a18cb280a29c15f6d0177b WatchSource:0}: Error finding container d42678f0309ff56a12f540d7469a670aeb0eccf0c1a18cb280a29c15f6d0177b: Status 404 returned error can't find the container with id d42678f0309ff56a12f540d7469a670aeb0eccf0c1a18cb280a29c15f6d0177b Oct 31 16:00:35.956331 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:35.956293 4865 generic.go:356] "Generic (PLEG): container finished" podID="55c9a085-d4a7-4874-a6f7-ac9e127bbefb" containerID="47c147679065b0abd35420a5a3653e607f8936cf1dc21676b7de2d253a1b02df" exitCode=0 Oct 31 16:00:35.956689 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:35.956358 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" event={"ID":"55c9a085-d4a7-4874-a6f7-ac9e127bbefb","Type":"ContainerDied","Data":"47c147679065b0abd35420a5a3653e607f8936cf1dc21676b7de2d253a1b02df"} Oct 31 16:00:35.956689 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:35.956441 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" event={"ID":"55c9a085-d4a7-4874-a6f7-ac9e127bbefb","Type":"ContainerStarted","Data":"d42678f0309ff56a12f540d7469a670aeb0eccf0c1a18cb280a29c15f6d0177b"} Oct 31 16:00:37.134029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.133997 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:37.164712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.164672 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-openshift-ca\") pod \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " Oct 31 16:00:37.164903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.164728 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-kube-api-access-bd9l9\" (UniqueName: \"kubernetes.io/projected/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-kube-api-access-bd9l9\") pod \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " Oct 31 16:00:37.164903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.164808 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-xks-ca\") pod \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\" (UID: \"55c9a085-d4a7-4874-a6f7-ac9e127bbefb\") " Oct 31 16:00:37.165143 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.165116 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-openshift-ca" pod "55c9a085-d4a7-4874-a6f7-ac9e127bbefb" (UID: "55c9a085-d4a7-4874-a6f7-ac9e127bbefb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:37.167124 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.167090 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-kube-api-access-bd9l9" pod "55c9a085-d4a7-4874-a6f7-ac9e127bbefb" (UID: "55c9a085-d4a7-4874-a6f7-ac9e127bbefb"). InnerVolumeSpecName "kube-api-access-bd9l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:37.167248 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.167141 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-xks-ca" pod "55c9a085-d4a7-4874-a6f7-ac9e127bbefb" (UID: "55c9a085-d4a7-4874-a6f7-ac9e127bbefb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:00:37.266526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.266431 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:37.266526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.266481 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd9l9\" (UniqueName: \"kubernetes.io/projected/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-kube-api-access-bd9l9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:37.266526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.266494 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c9a085-d4a7-4874-a6f7-ac9e127bbefb-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:37.460212 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.460170 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:37.473098 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.473063 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:37.971646 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.971614 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" Oct 31 16:00:37.971646 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.971640 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd" event={"ID":"55c9a085-d4a7-4874-a6f7-ac9e127bbefb","Type":"ContainerDied","Data":"d42678f0309ff56a12f540d7469a670aeb0eccf0c1a18cb280a29c15f6d0177b"} Oct 31 16:00:37.971907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.971687 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d42678f0309ff56a12f540d7469a670aeb0eccf0c1a18cb280a29c15f6d0177b" Oct 31 16:00:37.983427 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:37.983398 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:00:38.353395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:38.353316 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q"] Oct 31 16:00:38.361839 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:38.361813 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm"] Oct 31 16:00:38.438218 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:38.438183 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:39.547716 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:39.547681 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:42.409933 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:42.409899 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:44.419045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:44.419009 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:45.534980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:45.534949 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:47.026215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:47.026177 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:00:47.493436 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:47.493396 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:00:49.193230 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:49.193200 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:50.910557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:50.910516 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:51.027496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.027427 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8"] Oct 31 16:00:51.282540 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.282504 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:51.426977 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.426940 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:51.525926 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.525893 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x"] Oct 31 16:00:51.526529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.526515 4865 cpu_manager.go:414] "containerMap: removing container" podUID="55c9a085-d4a7-4874-a6f7-ac9e127bbefb" containerName="revoke-keys" Oct 31 16:00:51.526601 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.526530 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c9a085-d4a7-4874-a6f7-ac9e127bbefb" containerName="revoke-keys" Oct 31 16:00:51.526642 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.526628 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c9a085-d4a7-4874-a6f7-ac9e127bbefb" containerName="revoke-keys" Oct 31 16:00:51.530319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.530304 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.583887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.583831 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:00:51.584082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.583937 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:00:51.586368 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.586344 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x"] Oct 31 16:00:51.590798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.590773 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:00:51.590985 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.590967 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:00:51.666405 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.666375 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.666622 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.666418 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.666743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.666716 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xs9j\" (UniqueName: \"kubernetes.io/projected/74b30efe-778d-453b-aaf4-010f66fdcdf9-kube-api-access-5xs9j\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.768189 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.768157 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.768383 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.768195 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.768383 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.768251 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xs9j\" (UniqueName: \"kubernetes.io/projected/74b30efe-778d-453b-aaf4-010f66fdcdf9-kube-api-access-5xs9j\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.769089 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.768974 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.770778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.770754 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.827141 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.827068 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xs9j\" (UniqueName: \"kubernetes.io/projected/74b30efe-778d-453b-aaf4-010f66fdcdf9-kube-api-access-5xs9j\") pod \"maas-api-revoke-sub-cb24cc7f-d2r9x\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.843983 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.843955 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:51.966002 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:51.965967 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:52.221729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:52.221703 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x"] Oct 31 16:00:52.221874 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:00:52.221733 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b30efe_778d_453b_aaf4_010f66fdcdf9.slice/crio-1a1e6f3f24342b24f396df77e33d1ced6f9ea4cac78ff02dbe4a39e03faa2672 WatchSource:0}: Error finding container 1a1e6f3f24342b24f396df77e33d1ced6f9ea4cac78ff02dbe4a39e03faa2672: Status 404 returned error can't find the container with id 1a1e6f3f24342b24f396df77e33d1ced6f9ea4cac78ff02dbe4a39e03faa2672 Oct 31 16:00:53.074186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.074145 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" event={"ID":"74b30efe-778d-453b-aaf4-010f66fdcdf9","Type":"ContainerStarted","Data":"aaf1892a1f5751f7a79af36be38cbac7893e5bfb2842168cd3f368c04e4f36f9"} Oct 31 16:00:53.074627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.074202 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" event={"ID":"74b30efe-778d-453b-aaf4-010f66fdcdf9","Type":"ContainerStarted","Data":"1a1e6f3f24342b24f396df77e33d1ced6f9ea4cac78ff02dbe4a39e03faa2672"} Oct 31 16:00:53.189272 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.189211 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" podStartSLOduration=2.189198718 podStartE2EDuration="2.189198718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:53.188279233 +0000 UTC m=+4498.807830667" watchObservedRunningTime="2026-10-31 16:00:53.189198718 +0000 UTC m=+4498.808750128" Oct 31 16:00:53.375522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.375390 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" containerID="cri-o://68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d" gracePeriod=45 Oct 31 16:00:53.387962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.387927 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" containerID="cri-o://67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77" gracePeriod=45 Oct 31 16:00:53.669315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.669288 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 16:00:53.693545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.693520 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 16:00:53.796434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.796401 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tls-certs\") pod \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " Oct 31 16:00:53.796652 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.796493 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tmp-dir\") pod \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " Oct 31 16:00:53.796652 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.796617 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-dshm\") pod \"da378e66-4f18-415c-a334-4aeb5a33ce75\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " Oct 31 16:00:53.796788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.796690 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da378e66-4f18-415c-a334-4aeb5a33ce75-tls-certs\" (UniqueName: \"kubernetes.io/secret/da378e66-4f18-415c-a334-4aeb5a33ce75-tls-certs\") pod \"da378e66-4f18-415c-a334-4aeb5a33ce75\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " Oct 31 16:00:53.796788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.796730 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-model-cache\") pod \"da378e66-4f18-415c-a334-4aeb5a33ce75\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " Oct 31 16:00:53.796788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.796764 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-kube-api-access-89brt\" (UniqueName: \"kubernetes.io/projected/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-kube-api-access-89brt\") pod \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " Oct 31 16:00:53.796960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.796943 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-home\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-home\") pod \"da378e66-4f18-415c-a334-4aeb5a33ce75\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " Oct 31 16:00:53.797017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797001 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da378e66-4f18-415c-a334-4aeb5a33ce75-kube-api-access-hj6m4\" (UniqueName: \"kubernetes.io/projected/da378e66-4f18-415c-a334-4aeb5a33ce75-kube-api-access-hj6m4\") pod \"da378e66-4f18-415c-a334-4aeb5a33ce75\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " Oct 31 16:00:53.797077 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797064 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-model-cache\") pod \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " Oct 31 16:00:53.797154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797134 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-home\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-home\") pod \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " Oct 31 16:00:53.797221 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797170 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-dshm\") pod \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\" (UID: \"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e\") " Oct 31 16:00:53.797275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797248 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-tmp-dir\") pod \"da378e66-4f18-415c-a334-4aeb5a33ce75\" (UID: \"da378e66-4f18-415c-a334-4aeb5a33ce75\") " Oct 31 16:00:53.797523 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797451 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tmp-dir" pod "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" (UID: "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.797715 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797689 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-tmp-dir" pod "da378e66-4f18-415c-a334-4aeb5a33ce75" (UID: "da378e66-4f18-415c-a334-4aeb5a33ce75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.798033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797865 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-home" pod "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" (UID: "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.798033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.797898 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-model-cache" pod "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" (UID: "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.798244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.798062 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.798244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.798084 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.798244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.798098 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.798244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.798113 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.798937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.798487 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-model-cache" pod "da378e66-4f18-415c-a334-4aeb5a33ce75" (UID: "da378e66-4f18-415c-a334-4aeb5a33ce75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.798937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.798794 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-home" pod "da378e66-4f18-415c-a334-4aeb5a33ce75" (UID: "da378e66-4f18-415c-a334-4aeb5a33ce75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.801005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.800977 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da378e66-4f18-415c-a334-4aeb5a33ce75-kube-api-access-hj6m4" pod "da378e66-4f18-415c-a334-4aeb5a33ce75" (UID: "da378e66-4f18-415c-a334-4aeb5a33ce75"). InnerVolumeSpecName "kube-api-access-hj6m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:53.802511 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.801764 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-dshm" pod "da378e66-4f18-415c-a334-4aeb5a33ce75" (UID: "da378e66-4f18-415c-a334-4aeb5a33ce75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.802511 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.801826 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tls-certs" pod "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" (UID: "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:00:53.802511 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.801903 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-dshm" pod "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" (UID: "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:53.802511 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.802095 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da378e66-4f18-415c-a334-4aeb5a33ce75-tls-certs" pod "da378e66-4f18-415c-a334-4aeb5a33ce75" (UID: "da378e66-4f18-415c-a334-4aeb5a33ce75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:00:53.803451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.803428 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-kube-api-access-89brt" pod "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" (UID: "cbb1205e-6a6d-47a6-a09f-63b2f627ac1e"). InnerVolumeSpecName "kube-api-access-89brt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:53.899573 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899542 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.899573 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899571 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.899573 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899581 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da378e66-4f18-415c-a334-4aeb5a33ce75-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.899822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899593 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.899822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899605 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89brt\" (UniqueName: \"kubernetes.io/projected/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-kube-api-access-89brt\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.899822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899617 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da378e66-4f18-415c-a334-4aeb5a33ce75-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.899822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899628 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj6m4\" (UniqueName: \"kubernetes.io/projected/da378e66-4f18-415c-a334-4aeb5a33ce75-kube-api-access-hj6m4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:53.899822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:53.899641 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:54.083386 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.083350 4865 generic.go:356] "Generic (PLEG): container finished" podID="74b30efe-778d-453b-aaf4-010f66fdcdf9" containerID="aaf1892a1f5751f7a79af36be38cbac7893e5bfb2842168cd3f368c04e4f36f9" exitCode=0 Oct 31 16:00:54.083919 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.083432 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" event={"ID":"74b30efe-778d-453b-aaf4-010f66fdcdf9","Type":"ContainerDied","Data":"aaf1892a1f5751f7a79af36be38cbac7893e5bfb2842168cd3f368c04e4f36f9"} Oct 31 16:00:54.084848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.084825 4865 generic.go:356] "Generic (PLEG): container finished" podID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerID="68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d" exitCode=0 Oct 31 16:00:54.084974 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.084863 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" Oct 31 16:00:54.084974 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.084908 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" event={"ID":"da378e66-4f18-415c-a334-4aeb5a33ce75","Type":"ContainerDied","Data":"68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d"} Oct 31 16:00:54.084974 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.084947 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q" event={"ID":"da378e66-4f18-415c-a334-4aeb5a33ce75","Type":"ContainerDied","Data":"7319779579f661a91a30e925036f9e275c8f1813dd71af16ed700c7d8120bd98"} Oct 31 16:00:54.084974 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.084965 4865 scope.go:122] "RemoveContainer" containerID="68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d" Oct 31 16:00:54.086428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.086404 4865 generic.go:356] "Generic (PLEG): container finished" podID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerID="67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77" exitCode=0 Oct 31 16:00:54.086596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.086437 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" event={"ID":"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e","Type":"ContainerDied","Data":"67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77"} Oct 31 16:00:54.086596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.086455 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" Oct 31 16:00:54.086596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.086485 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" event={"ID":"cbb1205e-6a6d-47a6-a09f-63b2f627ac1e","Type":"ContainerDied","Data":"a693416de7ce41bc4ca6e26dc8d2bfdc07de43f515a1e98f9f34b1349d163989"} Oct 31 16:00:54.096878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.096852 4865 scope.go:122] "RemoveContainer" containerID="68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d" Oct 31 16:00:54.101232 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:54.101095 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d\": container with ID starting with 68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d not found: ID does not exist" containerID="68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d" Oct 31 16:00:54.101232 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.101135 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d"} err="failed to get container status \"68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d\": rpc error: code = NotFound desc = could not find container \"68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d\": container with ID starting with 68a78399f45db9c10011dbdb1cc6073f7b476e3ff974291199a8a30f01efc11d not found: ID does not exist" Oct 31 16:00:54.101232 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.101164 4865 scope.go:122] "RemoveContainer" containerID="67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77" Oct 31 16:00:54.110734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.110716 4865 scope.go:122] "RemoveContainer" containerID="67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77" Oct 31 16:00:54.111039 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:00:54.111014 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77\": container with ID starting with 67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77 not found: ID does not exist" containerID="67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77" Oct 31 16:00:54.111112 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.111050 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77"} err="failed to get container status \"67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77\": rpc error: code = NotFound desc = could not find container \"67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77\": container with ID starting with 67c909adc054ac48151cd7c2a0c031c2a0ccab431bf2fd6371abe326d67d5f77 not found: ID does not exist" Oct 31 16:00:54.330182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.330150 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q"] Oct 31 16:00:54.333096 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.333075 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/premium-w2-029d26-kserve-94b44467c-mcf4q"] Oct 31 16:00:54.445836 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.445756 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm"] Oct 31 16:00:54.452545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.452518 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm"] Oct 31 16:00:54.468944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.468916 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:54.497164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.497142 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" path="/var/lib/kubelet/pods/cbb1205e-6a6d-47a6-a09f-63b2f627ac1e/volumes" Oct 31 16:00:54.497681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.497666 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" path="/var/lib/kubelet/pods/da378e66-4f18-415c-a334-4aeb5a33ce75/volumes" Oct 31 16:00:54.664551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:54.664515 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/simulator-w2-029d26-kserve-84868b5d68-mfkzm" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:00:55.250866 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.250842 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:55.316820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.316795 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b30efe-778d-453b-aaf4-010f66fdcdf9-kube-api-access-5xs9j\" (UniqueName: \"kubernetes.io/projected/74b30efe-778d-453b-aaf4-010f66fdcdf9-kube-api-access-5xs9j\") pod \"74b30efe-778d-453b-aaf4-010f66fdcdf9\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " Oct 31 16:00:55.316975 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.316844 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-openshift-ca\") pod \"74b30efe-778d-453b-aaf4-010f66fdcdf9\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " Oct 31 16:00:55.316975 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.316880 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-xks-ca\") pod \"74b30efe-778d-453b-aaf4-010f66fdcdf9\" (UID: \"74b30efe-778d-453b-aaf4-010f66fdcdf9\") " Oct 31 16:00:55.317338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.317315 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-openshift-ca" pod "74b30efe-778d-453b-aaf4-010f66fdcdf9" (UID: "74b30efe-778d-453b-aaf4-010f66fdcdf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:55.319118 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.319092 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-xks-ca" pod "74b30efe-778d-453b-aaf4-010f66fdcdf9" (UID: "74b30efe-778d-453b-aaf4-010f66fdcdf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:00:55.319230 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.319126 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b30efe-778d-453b-aaf4-010f66fdcdf9-kube-api-access-5xs9j" pod "74b30efe-778d-453b-aaf4-010f66fdcdf9" (UID: "74b30efe-778d-453b-aaf4-010f66fdcdf9"). InnerVolumeSpecName "kube-api-access-5xs9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:55.418903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.418824 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:55.418903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.418853 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xs9j\" (UniqueName: \"kubernetes.io/projected/74b30efe-778d-453b-aaf4-010f66fdcdf9-kube-api-access-5xs9j\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:55.418903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:55.418866 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b30efe-778d-453b-aaf4-010f66fdcdf9-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:56.028762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:56.028719 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" podUID="37abe39b-a918-4e12-9b7b-59c499928838" containerName="payload-pre-processing" containerID="cri-o://d3b8f02f454f65f0612ce04ae6e61c4e25bfe03e41876b5fe6f6126b01a985d1" gracePeriod=25 Oct 31 16:00:56.103892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:56.103856 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" Oct 31 16:00:56.103892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:56.103885 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x" event={"ID":"74b30efe-778d-453b-aaf4-010f66fdcdf9","Type":"ContainerDied","Data":"1a1e6f3f24342b24f396df77e33d1ced6f9ea4cac78ff02dbe4a39e03faa2672"} Oct 31 16:00:56.104130 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:56.103939 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a1e6f3f24342b24f396df77e33d1ced6f9ea4cac78ff02dbe4a39e03faa2672" Oct 31 16:00:56.216922 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:56.216892 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:00:56.225148 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:56.225126 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:00:57.538202 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:57.538170 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:00:58.668641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:58.668612 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:00:58.677518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:58.677462 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:00:59.318710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:00:59.318679 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:01.641479 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:01.641434 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" podUID="37abe39b-a918-4e12-9b7b-59c499928838" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:01:04.015374 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:04.015338 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:04.409767 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:04.409682 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:04.984075 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:04.984047 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:08.045318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:08.045284 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8"] Oct 31 16:01:08.096126 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:08.096092 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a3df2e-kz4r8"] Oct 31 16:01:08.496854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:08.496815 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfc645f-fea4-4c6a-9e04-f4f384183abb" path="/var/lib/kubelet/pods/acfc645f-fea4-4c6a-9e04-f4f384183abb/volumes" Oct 31 16:01:08.689782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:08.689744 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:10.721983 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:10.721949 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:11.499755 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:11.499717 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:11.641434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:11.641392 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" podUID="37abe39b-a918-4e12-9b7b-59c499928838" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:01:11.800339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:11.800251 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:12.244160 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:12.244104 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:13.388791 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:13.388751 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:15.231870 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:15.231835 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:15.357241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:15.357206 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:16.238270 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.238218 4865 generic.go:356] "Generic (PLEG): container finished" podID="37abe39b-a918-4e12-9b7b-59c499928838" containerID="d3b8f02f454f65f0612ce04ae6e61c4e25bfe03e41876b5fe6f6126b01a985d1" exitCode=0 Oct 31 16:01:16.238770 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.238290 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" event={"ID":"37abe39b-a918-4e12-9b7b-59c499928838","Type":"ContainerDied","Data":"d3b8f02f454f65f0612ce04ae6e61c4e25bfe03e41876b5fe6f6126b01a985d1"} Oct 31 16:01:16.262005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.261970 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:16.299072 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.299021 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 16:01:16.368233 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.368204 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37abe39b-a918-4e12-9b7b-59c499928838-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37abe39b-a918-4e12-9b7b-59c499928838-plugins-config-volume\") pod \"37abe39b-a918-4e12-9b7b-59c499928838\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " Oct 31 16:01:16.368388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.368272 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37abe39b-a918-4e12-9b7b-59c499928838-kube-api-access-zbg6g\" (UniqueName: \"kubernetes.io/projected/37abe39b-a918-4e12-9b7b-59c499928838-kube-api-access-zbg6g\") pod \"37abe39b-a918-4e12-9b7b-59c499928838\" (UID: \"37abe39b-a918-4e12-9b7b-59c499928838\") " Oct 31 16:01:16.368679 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.368653 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37abe39b-a918-4e12-9b7b-59c499928838-plugins-config-volume" pod "37abe39b-a918-4e12-9b7b-59c499928838" (UID: "37abe39b-a918-4e12-9b7b-59c499928838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:01:16.370641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.370620 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37abe39b-a918-4e12-9b7b-59c499928838-kube-api-access-zbg6g" pod "37abe39b-a918-4e12-9b7b-59c499928838" (UID: "37abe39b-a918-4e12-9b7b-59c499928838"). InnerVolumeSpecName "kube-api-access-zbg6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:01:16.470355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.470317 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37abe39b-a918-4e12-9b7b-59c499928838-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:16.470355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:16.470352 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbg6g\" (UniqueName: \"kubernetes.io/projected/37abe39b-a918-4e12-9b7b-59c499928838-kube-api-access-zbg6g\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:17.245866 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:17.245836 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" Oct 31 16:01:17.246385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:17.245866 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8" event={"ID":"37abe39b-a918-4e12-9b7b-59c499928838","Type":"ContainerDied","Data":"5afedc8d2be13cc84ae9e6456f95aa99d9de8bd9790f679d763cc270be45a143"} Oct 31 16:01:17.246385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:17.245941 4865 scope.go:122] "RemoveContainer" containerID="d3b8f02f454f65f0612ce04ae6e61c4e25bfe03e41876b5fe6f6126b01a985d1" Oct 31 16:01:17.413414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:17.413377 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8"] Oct 31 16:01:17.464324 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:17.464294 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-029d26-574668f495-wdbd8"] Oct 31 16:01:17.491321 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:01:17.491289 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:01:18.098883 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:18.098848 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:18.496517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:18.496489 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37abe39b-a918-4e12-9b7b-59c499928838" path="/var/lib/kubelet/pods/37abe39b-a918-4e12-9b7b-59c499928838/volumes" Oct 31 16:01:21.893552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:21.893503 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:22.720267 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:22.720234 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:01:22.797403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:22.797372 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:27.442532 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:27.442461 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:27.565069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:27.565028 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:32.280267 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:32.280213 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:33.051360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:33.051325 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:36.022549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.022430 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk"] Oct 31 16:01:36.023241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023217 4865 cpu_manager.go:414] "containerMap: removing container" podUID="37abe39b-a918-4e12-9b7b-59c499928838" containerName="payload-pre-processing" Oct 31 16:01:36.023345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023243 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37abe39b-a918-4e12-9b7b-59c499928838" containerName="payload-pre-processing" Oct 31 16:01:36.023345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023267 4865 cpu_manager.go:414] "containerMap: removing container" podUID="74b30efe-778d-453b-aaf4-010f66fdcdf9" containerName="revoke-keys" Oct 31 16:01:36.023345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023277 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b30efe-778d-453b-aaf4-010f66fdcdf9" containerName="revoke-keys" Oct 31 16:01:36.023345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023298 4865 cpu_manager.go:414] "containerMap: removing container" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" Oct 31 16:01:36.023345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023305 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" Oct 31 16:01:36.023345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023347 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" Oct 31 16:01:36.023724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023356 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" Oct 31 16:01:36.023724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023510 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b30efe-778d-453b-aaf4-010f66fdcdf9" containerName="revoke-keys" Oct 31 16:01:36.023724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023527 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="da378e66-4f18-415c-a334-4aeb5a33ce75" containerName="main" Oct 31 16:01:36.023724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023545 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb1205e-6a6d-47a6-a09f-63b2f627ac1e" containerName="main" Oct 31 16:01:36.023724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.023559 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="37abe39b-a918-4e12-9b7b-59c499928838" containerName="payload-pre-processing" Oct 31 16:01:36.027026 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.027006 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.087780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.087745 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-31dff8\"/\"kube-root-ca.crt\"" Oct 31 16:01:36.087780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.087755 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-31dff8\"/\"openshift-service-ca.crt\"" Oct 31 16:01:36.139053 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.139015 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-31dff8\"/\"sub-test-model-31dff8-kserve-self-signed-certs\"" Oct 31 16:01:36.139053 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.139016 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-31dff8\"/\"default-dockercfg-wzsf5\"" Oct 31 16:01:36.142190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.142167 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk"] Oct 31 16:01:36.197272 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.197228 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:36.211888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.211851 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-home\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.211888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.211899 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvln\" (UniqueName: \"kubernetes.io/projected/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-kube-api-access-hjvln\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.212133 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.212039 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-model-cache\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.212190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.212163 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tmp-dir\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.212257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.212243 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tls-certs\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.212300 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.212273 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-dshm\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313083 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.312978 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-model-cache\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313085 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tmp-dir\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313160 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tls-certs\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313187 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-dshm\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313334 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-home\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313394 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvln\" (UniqueName: \"kubernetes.io/projected/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-kube-api-access-hjvln\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313441 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-model-cache\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313496 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tmp-dir\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.313753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.313733 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-home\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.315649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.315624 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-dshm\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.315954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.315937 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tls-certs\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.371204 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.371163 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvln\" (UniqueName: \"kubernetes.io/projected/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-kube-api-access-hjvln\") pod \"sub-test-model-31dff8-kserve-5c578496d5-l5crk\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.639800 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.639708 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:36.947230 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:36.947116 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:37.023491 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:37.022696 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk"] Oct 31 16:01:37.023491 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:01:37.023351 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4dcaf6b_ed2d_4c1b_afae_b4f195666170.slice/crio-431a2ed855bb25497068b8163179bba0e19ef82119ca8bfc05ba7217d1e4e4b1 WatchSource:0}: Error finding container 431a2ed855bb25497068b8163179bba0e19ef82119ca8bfc05ba7217d1e4e4b1: Status 404 returned error can't find the container with id 431a2ed855bb25497068b8163179bba0e19ef82119ca8bfc05ba7217d1e4e4b1 Oct 31 16:01:37.428921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:37.428870 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" event={"ID":"c4dcaf6b-ed2d-4c1b-afae-b4f195666170","Type":"ContainerStarted","Data":"dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2"} Oct 31 16:01:37.428921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:37.428923 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" event={"ID":"c4dcaf6b-ed2d-4c1b-afae-b4f195666170","Type":"ContainerStarted","Data":"431a2ed855bb25497068b8163179bba0e19ef82119ca8bfc05ba7217d1e4e4b1"} Oct 31 16:01:37.544051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:37.544000 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" podStartSLOduration=2.543986699 podStartE2EDuration="2.543986699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:01:37.543198142 +0000 UTC m=+4543.162749553" watchObservedRunningTime="2026-10-31 16:01:37.543986699 +0000 UTC m=+4543.163538111" Oct 31 16:01:37.640256 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:37.640227 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:41.580411 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:41.580377 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:46.210893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:46.210858 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:46.640218 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:46.640133 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:46.653400 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:46.653370 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:47.495438 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:01:47.495384 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:01:47.512191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:47.512158 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:01:50.320972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:50.320928 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:52.058818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:52.058785 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:52.743862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:52.743830 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:52.873148 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:52.873115 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:54.161105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:54.161072 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:54.616979 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:54.616949 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:54.628522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:54.628495 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22"] Oct 31 16:01:54.628752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:54.628727 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" containerID="cri-o://daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e" gracePeriod=30 Oct 31 16:01:56.062507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:56.062454 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:56.198017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:56.197981 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:56.984599 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:56.984567 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:01:59.915020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.914960 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 16:01:59.975776 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.975751 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e556e968-f2b7-4b38-bc57-e2f6216260e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e556e968-f2b7-4b38-bc57-e2f6216260e7-istiod-ca-cert\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.975924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.975796 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-podinfo\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.975924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.975835 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-token\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-token\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.975924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.975903 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-certs\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.976098 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.975970 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-envoy\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.976180 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976154 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-kube-api-access-w2sj5\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-kube-api-access-w2sj5\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.976250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976182 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e556e968-f2b7-4b38-bc57-e2f6216260e7-istiod-ca-cert" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:01:59.976250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976207 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-socket\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.976368 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976267 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-credential-socket\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.976368 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976296 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-data\") pod \"e556e968-f2b7-4b38-bc57-e2f6216260e7\" (UID: \"e556e968-f2b7-4b38-bc57-e2f6216260e7\") " Oct 31 16:01:59.976676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976290 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-certs" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:01:59.976676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976515 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-data" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:01:59.976676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976527 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-credential-socket" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:01:59.976676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.976657 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-socket" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:01:59.977191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.977167 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e556e968-f2b7-4b38-bc57-e2f6216260e7-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:59.977254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.977200 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:59.977254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.977216 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:59.977254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.977231 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:59.977254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.977246 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:59.978621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.978593 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-token" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:01:59.978743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.978723 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-kube-api-access-w2sj5" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "kube-api-access-w2sj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:01:59.978977 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.978961 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-envoy" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:01:59.979263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:01:59.979243 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-podinfo" pod "e556e968-f2b7-4b38-bc57-e2f6216260e7" (UID: "e556e968-f2b7-4b38-bc57-e2f6216260e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:02:00.078853 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.078815 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.078853 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.078849 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2sj5\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-kube-api-access-w2sj5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.078853 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.078860 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.079085 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.078870 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e556e968-f2b7-4b38-bc57-e2f6216260e7-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.586876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.586842 4865 generic.go:356] "Generic (PLEG): container finished" podID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerID="daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e" exitCode=0 Oct 31 16:02:00.587071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.586884 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" Oct 31 16:02:00.587071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.586936 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" event={"ID":"e556e968-f2b7-4b38-bc57-e2f6216260e7","Type":"ContainerDied","Data":"daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e"} Oct 31 16:02:00.587071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.586986 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22" event={"ID":"e556e968-f2b7-4b38-bc57-e2f6216260e7","Type":"ContainerDied","Data":"68812e23f1b34de69bb4e0c0ae55505217025d82d531665aa2aa6c3a1d1b7617"} Oct 31 16:02:00.587071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.587010 4865 scope.go:122] "RemoveContainer" containerID="daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e" Oct 31 16:02:00.597376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.597354 4865 scope.go:122] "RemoveContainer" containerID="daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e" Oct 31 16:02:00.597654 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:02:00.597629 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e\": container with ID starting with daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e not found: ID does not exist" containerID="daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e" Oct 31 16:02:00.597771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.597660 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e"} err="failed to get container status \"daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e\": rpc error: code = NotFound desc = could not find container \"daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e\": container with ID starting with daa56002ad093038023e0c7ef792811add4593080cd39a96bb74d9bda1cf363e not found: ID does not exist" Oct 31 16:02:00.757425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.757344 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22"] Oct 31 16:02:00.806753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:00.806719 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-t5j22"] Oct 31 16:02:01.736509 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:01.736447 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:02.496441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:02.496411 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" path="/var/lib/kubelet/pods/e556e968-f2b7-4b38-bc57-e2f6216260e7/volumes" Oct 31 16:02:03.603063 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:03.603029 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:02:06.395618 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:06.395585 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:11.089360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:11.089327 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:11.271364 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:11.271330 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:12.609490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:12.609439 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:15.802147 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:15.802106 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:17.489953 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:02:17.489910 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:02:20.486997 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:20.486961 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:24.873232 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:24.873196 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:02:25.191186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:25.191101 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:29.907076 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:29.907038 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:34.008797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.008763 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl"] Oct 31 16:02:34.009382 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.009367 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" Oct 31 16:02:34.009434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.009384 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" Oct 31 16:02:34.009522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.009509 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e556e968-f2b7-4b38-bc57-e2f6216260e7" containerName="istio-proxy" Oct 31 16:02:34.012705 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.012691 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.068108 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.068080 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl"] Oct 31 16:02:34.115216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.115188 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.115373 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.115222 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.116903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.116879 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:02:34.117174 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.117160 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:02:34.165913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.165884 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.166058 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.166001 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.166102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.166075 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9ms\" (UniqueName: \"kubernetes.io/projected/1c771321-426c-4a86-9f12-6ccf496115a8-kube-api-access-5m9ms\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.267152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.267075 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9ms\" (UniqueName: \"kubernetes.io/projected/1c771321-426c-4a86-9f12-6ccf496115a8-kube-api-access-5m9ms\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.267311 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.267203 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.267311 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.267243 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.267932 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.267899 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.269800 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.269765 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.325034 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.325004 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9ms\" (UniqueName: \"kubernetes.io/projected/1c771321-426c-4a86-9f12-6ccf496115a8-kube-api-access-5m9ms\") pod \"maas-api-revoke-sub-3726ba4a-69djl\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.326628 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.326608 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:34.599571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.599500 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:34.704843 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.704813 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl"] Oct 31 16:02:34.823792 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:34.823763 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" event={"ID":"1c771321-426c-4a86-9f12-6ccf496115a8","Type":"ContainerStarted","Data":"576ed6e898dd2194f81d584a4baa16124012701ed016d07149c100897343892e"} Oct 31 16:02:35.832482 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:35.832370 4865 generic.go:356] "Generic (PLEG): container finished" podID="1c771321-426c-4a86-9f12-6ccf496115a8" containerID="8fb146b2d052bbba45f5b3c0dc5964d888c342d23cf219996ed7864635aa8716" exitCode=0 Oct 31 16:02:35.832891 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:35.832460 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" event={"ID":"1c771321-426c-4a86-9f12-6ccf496115a8","Type":"ContainerDied","Data":"8fb146b2d052bbba45f5b3c0dc5964d888c342d23cf219996ed7864635aa8716"} Oct 31 16:02:37.006114 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.006088 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:37.100707 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.100640 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-openshift-ca\") pod \"1c771321-426c-4a86-9f12-6ccf496115a8\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " Oct 31 16:02:37.100849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.100716 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c771321-426c-4a86-9f12-6ccf496115a8-kube-api-access-5m9ms\" (UniqueName: \"kubernetes.io/projected/1c771321-426c-4a86-9f12-6ccf496115a8-kube-api-access-5m9ms\") pod \"1c771321-426c-4a86-9f12-6ccf496115a8\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " Oct 31 16:02:37.100849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.100783 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-xks-ca\") pod \"1c771321-426c-4a86-9f12-6ccf496115a8\" (UID: \"1c771321-426c-4a86-9f12-6ccf496115a8\") " Oct 31 16:02:37.101042 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.101017 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-openshift-ca" pod "1c771321-426c-4a86-9f12-6ccf496115a8" (UID: "1c771321-426c-4a86-9f12-6ccf496115a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:02:37.101182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.101167 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:37.103006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.102977 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c771321-426c-4a86-9f12-6ccf496115a8-kube-api-access-5m9ms" pod "1c771321-426c-4a86-9f12-6ccf496115a8" (UID: "1c771321-426c-4a86-9f12-6ccf496115a8"). InnerVolumeSpecName "kube-api-access-5m9ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:02:37.103117 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.103016 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-xks-ca" pod "1c771321-426c-4a86-9f12-6ccf496115a8" (UID: "1c771321-426c-4a86-9f12-6ccf496115a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:02:37.202639 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.202603 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9ms\" (UniqueName: \"kubernetes.io/projected/1c771321-426c-4a86-9f12-6ccf496115a8-kube-api-access-5m9ms\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:37.202639 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.202633 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c771321-426c-4a86-9f12-6ccf496115a8-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:37.846561 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.846515 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" event={"ID":"1c771321-426c-4a86-9f12-6ccf496115a8","Type":"ContainerDied","Data":"576ed6e898dd2194f81d584a4baa16124012701ed016d07149c100897343892e"} Oct 31 16:02:37.846749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.846582 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="576ed6e898dd2194f81d584a4baa16124012701ed016d07149c100897343892e" Oct 31 16:02:37.846749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:37.846531 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl" Oct 31 16:02:39.347273 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:39.347242 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:44.049198 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.049159 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w"] Oct 31 16:02:44.055039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.050189 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:44.096794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.096761 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369e434a-kzr8w"] Oct 31 16:02:44.496503 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.496448 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57b8bb7-dd26-49c7-b98d-d8e4afec0b28" path="/var/lib/kubelet/pods/f57b8bb7-dd26-49c7-b98d-d8e4afec0b28/volumes" Oct 31 16:02:44.682778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.682750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:44.713068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.713038 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2"] Oct 31 16:02:44.713931 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.713909 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1c771321-426c-4a86-9f12-6ccf496115a8" containerName="revoke-keys" Oct 31 16:02:44.713991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.713935 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c771321-426c-4a86-9f12-6ccf496115a8" containerName="revoke-keys" Oct 31 16:02:44.714172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.714158 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c771321-426c-4a86-9f12-6ccf496115a8" containerName="revoke-keys" Oct 31 16:02:44.718356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.718335 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.772040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.771953 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2"] Oct 31 16:02:44.817969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.817939 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:02:44.818141 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.817939 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:02:44.818141 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.818134 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:02:44.822425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.822410 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:02:44.891439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.891413 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbpx\" (UniqueName: \"kubernetes.io/projected/012106d2-3d26-44c9-8df2-d461d1307801-kube-api-access-xvbpx\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.891626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.891592 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.891687 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.891627 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.992834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.992803 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.993125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.992845 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.993125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.992934 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbpx\" (UniqueName: \"kubernetes.io/projected/012106d2-3d26-44c9-8df2-d461d1307801-kube-api-access-xvbpx\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.993579 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.993561 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:44.995613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:44.995584 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:45.078328 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:45.078232 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbpx\" (UniqueName: \"kubernetes.io/projected/012106d2-3d26-44c9-8df2-d461d1307801-kube-api-access-xvbpx\") pod \"maas-api-revoke-sub-e721021a-v9gt2\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:45.333572 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:45.333482 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:45.707175 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:02:45.707141 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012106d2_3d26_44c9_8df2_d461d1307801.slice/crio-6940622897ac27da5148651e0155a77a8058c6206fac76458933a88462141e74 WatchSource:0}: Error finding container 6940622897ac27da5148651e0155a77a8058c6206fac76458933a88462141e74: Status 404 returned error can't find the container with id 6940622897ac27da5148651e0155a77a8058c6206fac76458933a88462141e74 Oct 31 16:02:45.707416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:45.707394 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2"] Oct 31 16:02:45.899302 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:45.899252 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" event={"ID":"012106d2-3d26-44c9-8df2-d461d1307801","Type":"ContainerStarted","Data":"6940622897ac27da5148651e0155a77a8058c6206fac76458933a88462141e74"} Oct 31 16:02:46.907097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:46.907012 4865 generic.go:356] "Generic (PLEG): container finished" podID="012106d2-3d26-44c9-8df2-d461d1307801" containerID="afd407f9214e3d99777ab538f1d25e8c22d877d927bbe140c691549e1990600f" exitCode=0 Oct 31 16:02:46.907492 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:46.907098 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" event={"ID":"012106d2-3d26-44c9-8df2-d461d1307801","Type":"ContainerDied","Data":"afd407f9214e3d99777ab538f1d25e8c22d877d927bbe140c691549e1990600f"} Oct 31 16:02:47.491296 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:02:47.491257 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:02:48.062710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.062682 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:48.136421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.136392 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012106d2-3d26-44c9-8df2-d461d1307801-kube-api-access-xvbpx\" (UniqueName: \"kubernetes.io/projected/012106d2-3d26-44c9-8df2-d461d1307801-kube-api-access-xvbpx\") pod \"012106d2-3d26-44c9-8df2-d461d1307801\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " Oct 31 16:02:48.136586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.136433 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-xks-ca\") pod \"012106d2-3d26-44c9-8df2-d461d1307801\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " Oct 31 16:02:48.136586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.136561 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-openshift-ca\") pod \"012106d2-3d26-44c9-8df2-d461d1307801\" (UID: \"012106d2-3d26-44c9-8df2-d461d1307801\") " Oct 31 16:02:48.136915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.136891 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-openshift-ca" pod "012106d2-3d26-44c9-8df2-d461d1307801" (UID: "012106d2-3d26-44c9-8df2-d461d1307801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:02:48.137105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.137088 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:48.138731 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.138711 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-xks-ca" pod "012106d2-3d26-44c9-8df2-d461d1307801" (UID: "012106d2-3d26-44c9-8df2-d461d1307801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:02:48.138839 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.138820 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012106d2-3d26-44c9-8df2-d461d1307801-kube-api-access-xvbpx" pod "012106d2-3d26-44c9-8df2-d461d1307801" (UID: "012106d2-3d26-44c9-8df2-d461d1307801"). InnerVolumeSpecName "kube-api-access-xvbpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:02:48.238007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.237971 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbpx\" (UniqueName: \"kubernetes.io/projected/012106d2-3d26-44c9-8df2-d461d1307801-kube-api-access-xvbpx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:48.238007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.238003 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012106d2-3d26-44c9-8df2-d461d1307801-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:48.781181 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.781092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:48.920712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.920678 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" event={"ID":"012106d2-3d26-44c9-8df2-d461d1307801","Type":"ContainerDied","Data":"6940622897ac27da5148651e0155a77a8058c6206fac76458933a88462141e74"} Oct 31 16:02:48.920712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.920703 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2" Oct 31 16:02:48.920923 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:48.920729 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6940622897ac27da5148651e0155a77a8058c6206fac76458933a88462141e74" Oct 31 16:02:50.045081 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:50.045047 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj"] Oct 31 16:02:50.096040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:50.096006 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8755aabf-5dtgj"] Oct 31 16:02:50.502850 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:50.502809 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2ea294-1ecd-4d79-b1ba-63f480460945" path="/var/lib/kubelet/pods/bb2ea294-1ecd-4d79-b1ba-63f480460945/volumes" Oct 31 16:02:53.407426 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:53.407386 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:55.312669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:55.312632 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:55.459558 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:55.459522 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:58.053020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:58.052987 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn"] Oct 31 16:02:58.091121 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:58.091092 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c468c833-9j8qn"] Oct 31 16:02:58.118248 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:58.118206 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:58.496773 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:58.496740 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d238adf-64a6-4e2d-9872-aa7d0ffa54fa" path="/var/lib/kubelet/pods/8d238adf-64a6-4e2d-9872-aa7d0ffa54fa/volumes" Oct 31 16:02:59.930320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:02:59.930271 4865 scope.go:122] "RemoveContainer" containerID="893642d7c951dab3dbd4fe2f722a4a2c0a2a6e1575ec6fea9fca236d6f695f01" Oct 31 16:03:02.781053 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:02.781017 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:05.587676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:05.587588 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:03:07.594156 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:07.594118 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:11.438373 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:11.438337 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:12.101742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:12.101707 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:13.037678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:13.037645 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:16.805055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:16.805025 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:16.955007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:16.954974 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:17.489780 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:03:17.489747 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:03:21.194457 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:21.194421 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp"] Oct 31 16:03:21.195314 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:21.195284 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:21.503732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:21.503702 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:26.189832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:26.189798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:30.796122 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:30.796082 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:30.931608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:30.931556 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:32.746928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:32.746884 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:32.752056 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:32.752029 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7"] Oct 31 16:03:33.260255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:33.260211 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:35.622391 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:35.622360 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:36.215095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.215036 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" podUID="112f74aa-ba8c-4799-92a7-3a06e792d4bf" containerName="main" containerID="cri-o://070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c" gracePeriod=45 Oct 31 16:03:36.322966 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.322929 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" podUID="112f74aa-ba8c-4799-92a7-3a06e792d4bf" containerName="main" probeResult="failure" output="Get \"https://10.132.1.105:8000/ready\": dial tcp 10.132.1.105:8000: connect: connection refused" Oct 31 16:03:36.487717 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.487692 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 16:03:36.512165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512140 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-model-cache\") pod \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " Oct 31 16:03:36.512319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512267 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112f74aa-ba8c-4799-92a7-3a06e792d4bf-kube-api-access-68vbm\" (UniqueName: \"kubernetes.io/projected/112f74aa-ba8c-4799-92a7-3a06e792d4bf-kube-api-access-68vbm\") pod \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " Oct 31 16:03:36.512319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512296 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tls-certs\") pod \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " Oct 31 16:03:36.512437 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512338 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-home\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-home\") pod \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " Oct 31 16:03:36.512437 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512377 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-model-cache" pod "112f74aa-ba8c-4799-92a7-3a06e792d4bf" (UID: "112f74aa-ba8c-4799-92a7-3a06e792d4bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:36.512437 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512404 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tmp-dir\") pod \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " Oct 31 16:03:36.512669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512500 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-dshm\") pod \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\" (UID: \"112f74aa-ba8c-4799-92a7-3a06e792d4bf\") " Oct 31 16:03:36.512669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512622 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-home" pod "112f74aa-ba8c-4799-92a7-3a06e792d4bf" (UID: "112f74aa-ba8c-4799-92a7-3a06e792d4bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:36.512669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512641 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tmp-dir" pod "112f74aa-ba8c-4799-92a7-3a06e792d4bf" (UID: "112f74aa-ba8c-4799-92a7-3a06e792d4bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:36.513152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.512998 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:36.513152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.513124 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:36.513152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.513141 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:36.514699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.514668 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tls-certs" pod "112f74aa-ba8c-4799-92a7-3a06e792d4bf" (UID: "112f74aa-ba8c-4799-92a7-3a06e792d4bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:03:36.514699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.514679 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-dshm" pod "112f74aa-ba8c-4799-92a7-3a06e792d4bf" (UID: "112f74aa-ba8c-4799-92a7-3a06e792d4bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:36.514839 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.514786 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112f74aa-ba8c-4799-92a7-3a06e792d4bf-kube-api-access-68vbm" pod "112f74aa-ba8c-4799-92a7-3a06e792d4bf" (UID: "112f74aa-ba8c-4799-92a7-3a06e792d4bf"). InnerVolumeSpecName "kube-api-access-68vbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:03:36.615192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.615153 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/112f74aa-ba8c-4799-92a7-3a06e792d4bf-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:36.615192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.615195 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vbm\" (UniqueName: \"kubernetes.io/projected/112f74aa-ba8c-4799-92a7-3a06e792d4bf-kube-api-access-68vbm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:36.615451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:36.615212 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112f74aa-ba8c-4799-92a7-3a06e792d4bf-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.233009 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.232970 4865 generic.go:356] "Generic (PLEG): container finished" podID="112f74aa-ba8c-4799-92a7-3a06e792d4bf" containerID="070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c" exitCode=0 Oct 31 16:03:37.233432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.233037 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" Oct 31 16:03:37.233432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.233061 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" event={"ID":"112f74aa-ba8c-4799-92a7-3a06e792d4bf","Type":"ContainerDied","Data":"070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c"} Oct 31 16:03:37.233432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.233110 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp" event={"ID":"112f74aa-ba8c-4799-92a7-3a06e792d4bf","Type":"ContainerDied","Data":"8f330c968cfc7d2e5741b175e931e3abc6db333c85f611d01ac8dd7ae8b3ecbe"} Oct 31 16:03:37.233432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.233127 4865 scope.go:122] "RemoveContainer" containerID="070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c" Oct 31 16:03:37.244215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.244199 4865 scope.go:122] "RemoveContainer" containerID="070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c" Oct 31 16:03:37.244520 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:03:37.244496 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c\": container with ID starting with 070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c not found: ID does not exist" containerID="070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c" Oct 31 16:03:37.244600 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.244530 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c"} err="failed to get container status \"070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c\": rpc error: code = NotFound desc = could not find container \"070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c\": container with ID starting with 070831a16946832a007f5fcad6bb020ed45d656e50a22e7cc4bc35cfcf7c179c not found: ID does not exist" Oct 31 16:03:37.399793 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.399757 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp"] Oct 31 16:03:37.450594 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:37.450554 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-531c3c/rate-test-model-531c3c-kserve-6bf985d8c7-rldcp"] Oct 31 16:03:38.496786 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:38.496754 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112f74aa-ba8c-4799-92a7-3a06e792d4bf" path="/var/lib/kubelet/pods/112f74aa-ba8c-4799-92a7-3a06e792d4bf/volumes" Oct 31 16:03:39.188490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:39.188431 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:40.401972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:40.401936 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:40.611100 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:40.611057 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn"] Oct 31 16:03:40.714968 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:40.714927 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx"] Oct 31 16:03:40.954739 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:40.954675 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:41.079609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:41.079520 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:44.628701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:44.628648 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:45.323384 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:45.323354 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:45.611815 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:45.611693 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" podUID="c7049ebd-d545-4708-aec0-77058dd16d15" containerName="payload-processing" containerID="cri-o://5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d" gracePeriod=25 Oct 31 16:03:45.715729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:45.715684 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" podUID="f9c23a96-8377-428b-942a-8bada9964aba" containerName="payload-pre-processing" containerID="cri-o://fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0" gracePeriod=25 Oct 31 16:03:47.491011 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:03:47.490976 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:03:47.772960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:47.772917 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" podUID="ab2c6505-e58e-40b6-9f64-5236f9dd5035" containerName="main" containerID="cri-o://833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db" gracePeriod=45 Oct 31 16:03:48.045252 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.045218 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:03:48.171089 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171054 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tls-certs\") pod \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " Oct 31 16:03:48.171291 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171154 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2c6505-e58e-40b6-9f64-5236f9dd5035-kube-api-access-5fb26\" (UniqueName: \"kubernetes.io/projected/ab2c6505-e58e-40b6-9f64-5236f9dd5035-kube-api-access-5fb26\") pod \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " Oct 31 16:03:48.171291 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171253 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-home\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-home\") pod \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " Oct 31 16:03:48.171436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171298 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-dshm\") pod \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " Oct 31 16:03:48.171436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171344 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tmp-dir\") pod \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " Oct 31 16:03:48.171592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171455 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-model-cache\") pod \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\" (UID: \"ab2c6505-e58e-40b6-9f64-5236f9dd5035\") " Oct 31 16:03:48.171592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171538 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-home" pod "ab2c6505-e58e-40b6-9f64-5236f9dd5035" (UID: "ab2c6505-e58e-40b6-9f64-5236f9dd5035"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:48.172003 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171712 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tmp-dir" pod "ab2c6505-e58e-40b6-9f64-5236f9dd5035" (UID: "ab2c6505-e58e-40b6-9f64-5236f9dd5035"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:48.172003 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171807 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-model-cache" pod "ab2c6505-e58e-40b6-9f64-5236f9dd5035" (UID: "ab2c6505-e58e-40b6-9f64-5236f9dd5035"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:48.172003 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171972 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:48.172003 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171988 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:48.172003 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.171998 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:48.174104 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.174068 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tls-certs" pod "ab2c6505-e58e-40b6-9f64-5236f9dd5035" (UID: "ab2c6505-e58e-40b6-9f64-5236f9dd5035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:03:48.174365 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.174343 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-dshm" pod "ab2c6505-e58e-40b6-9f64-5236f9dd5035" (UID: "ab2c6505-e58e-40b6-9f64-5236f9dd5035"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:48.174460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.174395 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2c6505-e58e-40b6-9f64-5236f9dd5035-kube-api-access-5fb26" pod "ab2c6505-e58e-40b6-9f64-5236f9dd5035" (UID: "ab2c6505-e58e-40b6-9f64-5236f9dd5035"). InnerVolumeSpecName "kube-api-access-5fb26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:03:48.273390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.273343 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fb26\" (UniqueName: \"kubernetes.io/projected/ab2c6505-e58e-40b6-9f64-5236f9dd5035-kube-api-access-5fb26\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:48.273390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.273384 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab2c6505-e58e-40b6-9f64-5236f9dd5035-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:48.273390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.273394 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab2c6505-e58e-40b6-9f64-5236f9dd5035-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:03:48.304019 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.303924 4865 generic.go:356] "Generic (PLEG): container finished" podID="ab2c6505-e58e-40b6-9f64-5236f9dd5035" containerID="833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db" exitCode=0 Oct 31 16:03:48.304019 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.303969 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" Oct 31 16:03:48.304019 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.304008 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" event={"ID":"ab2c6505-e58e-40b6-9f64-5236f9dd5035","Type":"ContainerDied","Data":"833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db"} Oct 31 16:03:48.304309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.304053 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" event={"ID":"ab2c6505-e58e-40b6-9f64-5236f9dd5035","Type":"ContainerDied","Data":"6cf95a87280da5f24fcc3a50d37150a6a9444b7a73adc2c3da600258a1660f79"} Oct 31 16:03:48.304309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.304073 4865 scope.go:122] "RemoveContainer" containerID="833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db" Oct 31 16:03:48.315415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.315395 4865 scope.go:122] "RemoveContainer" containerID="833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db" Oct 31 16:03:48.315770 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:03:48.315748 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db\": container with ID starting with 833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db not found: ID does not exist" containerID="833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db" Oct 31 16:03:48.315851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.315778 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db"} err="failed to get container status \"833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db\": rpc error: code = NotFound desc = could not find container \"833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db\": container with ID starting with 833f79410af97b1a116c0f87abc1d7182f0fd6a6faf1b903ee076b56fde948db not found: ID does not exist" Oct 31 16:03:48.472618 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.472579 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7"] Oct 31 16:03:48.477036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.477007 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7"] Oct 31 16:03:48.486267 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.486231 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:03:48.497263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.497234 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2c6505-e58e-40b6-9f64-5236f9dd5035" path="/var/lib/kubelet/pods/ab2c6505-e58e-40b6-9f64-5236f9dd5035/volumes" Oct 31 16:03:48.973208 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:48.973153 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-55b3fb/rate-test-model-55b3fb-kserve-67f86dd49-h5qf7" podUID="ab2c6505-e58e-40b6-9f64-5236f9dd5035" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:03:49.992248 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:49.992212 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:50.742029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:50.741988 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" podUID="f9c23a96-8377-428b-942a-8bada9964aba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:03:50.742273 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:50.741999 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" podUID="c7049ebd-d545-4708-aec0-77058dd16d15" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:03:54.796970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:54.796936 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:59.029670 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.029636 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w"] Oct 31 16:03:59.030206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030193 4865 cpu_manager.go:414] "containerMap: removing container" podUID="012106d2-3d26-44c9-8df2-d461d1307801" containerName="revoke-keys" Oct 31 16:03:59.030263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030207 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012106d2-3d26-44c9-8df2-d461d1307801" containerName="revoke-keys" Oct 31 16:03:59.030263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030222 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ab2c6505-e58e-40b6-9f64-5236f9dd5035" containerName="main" Oct 31 16:03:59.030263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030228 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2c6505-e58e-40b6-9f64-5236f9dd5035" containerName="main" Oct 31 16:03:59.030263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030258 4865 cpu_manager.go:414] "containerMap: removing container" podUID="112f74aa-ba8c-4799-92a7-3a06e792d4bf" containerName="main" Oct 31 16:03:59.030263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030263 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112f74aa-ba8c-4799-92a7-3a06e792d4bf" containerName="main" Oct 31 16:03:59.030521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030353 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="112f74aa-ba8c-4799-92a7-3a06e792d4bf" containerName="main" Oct 31 16:03:59.030521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030367 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="012106d2-3d26-44c9-8df2-d461d1307801" containerName="revoke-keys" Oct 31 16:03:59.030521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.030376 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2c6505-e58e-40b6-9f64-5236f9dd5035" containerName="main" Oct 31 16:03:59.034961 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.034946 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.089532 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.089495 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w"] Oct 31 16:03:59.109852 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.109822 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwtj\" (UniqueName: \"kubernetes.io/projected/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-kube-api-access-7kwtj\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.110022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.109955 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.110022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.110014 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.135991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.135956 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:03:59.135991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.135964 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:03:59.136171 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.136053 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:03:59.139008 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.138989 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:03:59.210852 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.210816 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.211079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.210953 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwtj\" (UniqueName: \"kubernetes.io/projected/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-kube-api-access-7kwtj\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.211079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.211021 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.211590 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.211566 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.213588 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.213568 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.270274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.270245 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwtj\" (UniqueName: \"kubernetes.io/projected/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-kube-api-access-7kwtj\") pod \"maas-api-revoke-sub-ae4b3099-xxf7w\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.348697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.348615 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:03:59.509757 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.509720 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:59.724513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.723788 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w"] Oct 31 16:03:59.724513 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:03:59.724027 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fea8f1_a4d4_4b15_ae66_77fcf02ec623.slice/crio-17d28b0d0ced3e6aa44d3a402e16cb37cf4e532b8143c8c97269de7f4a2cdf7c WatchSource:0}: Error finding container 17d28b0d0ced3e6aa44d3a402e16cb37cf4e532b8143c8c97269de7f4a2cdf7c: Status 404 returned error can't find the container with id 17d28b0d0ced3e6aa44d3a402e16cb37cf4e532b8143c8c97269de7f4a2cdf7c Oct 31 16:03:59.959880 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.959843 4865 scope.go:122] "RemoveContainer" containerID="38112de63ab1c6433d50d8145eaa81df7783555469e5406c6e2d9c66579849ee" Oct 31 16:03:59.970364 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.970343 4865 scope.go:122] "RemoveContainer" containerID="ddc4d318d14600af09da459ef3a60e4328e45937d7d4f510565ba557fc8a5826" Oct 31 16:03:59.980106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:03:59.980082 4865 scope.go:122] "RemoveContainer" containerID="64630c79c771d0b65950ca8b363e80134a4cd6ccf90a9d18c10bcb8c9c5fccb7" Oct 31 16:04:00.403374 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:00.403288 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" event={"ID":"53fea8f1-a4d4-4b15-ae66-77fcf02ec623","Type":"ContainerStarted","Data":"8ff1bd947ab979d3c5dcc7c6f2caeb8de5774f621f0308c23506f1579d710c38"} Oct 31 16:04:00.403374 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:00.403338 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" event={"ID":"53fea8f1-a4d4-4b15-ae66-77fcf02ec623","Type":"ContainerStarted","Data":"17d28b0d0ced3e6aa44d3a402e16cb37cf4e532b8143c8c97269de7f4a2cdf7c"} Oct 31 16:04:00.742714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:00.742673 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" podUID="c7049ebd-d545-4708-aec0-77058dd16d15" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:00.742914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:00.742691 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" podUID="f9c23a96-8377-428b-942a-8bada9964aba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:01.411506 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:01.411456 4865 generic.go:356] "Generic (PLEG): container finished" podID="53fea8f1-a4d4-4b15-ae66-77fcf02ec623" containerID="8ff1bd947ab979d3c5dcc7c6f2caeb8de5774f621f0308c23506f1579d710c38" exitCode=0 Oct 31 16:04:01.411972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:01.411540 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" event={"ID":"53fea8f1-a4d4-4b15-ae66-77fcf02ec623","Type":"ContainerDied","Data":"8ff1bd947ab979d3c5dcc7c6f2caeb8de5774f621f0308c23506f1579d710c38"} Oct 31 16:04:02.576407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.576379 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:04:02.653418 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.653389 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-openshift-ca\") pod \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " Oct 31 16:04:02.653616 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.653442 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-xks-ca\") pod \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " Oct 31 16:04:02.653755 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.653730 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-kube-api-access-7kwtj\" (UniqueName: \"kubernetes.io/projected/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-kube-api-access-7kwtj\") pod \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\" (UID: \"53fea8f1-a4d4-4b15-ae66-77fcf02ec623\") " Oct 31 16:04:02.653880 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.653799 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-openshift-ca" pod "53fea8f1-a4d4-4b15-ae66-77fcf02ec623" (UID: "53fea8f1-a4d4-4b15-ae66-77fcf02ec623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:02.654134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.654110 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:02.655664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.655639 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-xks-ca" pod "53fea8f1-a4d4-4b15-ae66-77fcf02ec623" (UID: "53fea8f1-a4d4-4b15-ae66-77fcf02ec623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:04:02.655896 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.655869 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-kube-api-access-7kwtj" pod "53fea8f1-a4d4-4b15-ae66-77fcf02ec623" (UID: "53fea8f1-a4d4-4b15-ae66-77fcf02ec623"). InnerVolumeSpecName "kube-api-access-7kwtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:02.755791 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.755753 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwtj\" (UniqueName: \"kubernetes.io/projected/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-kube-api-access-7kwtj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:02.755791 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:02.755788 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53fea8f1-a4d4-4b15-ae66-77fcf02ec623-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:03.427174 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:03.427142 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" Oct 31 16:04:03.427356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:03.427175 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w" event={"ID":"53fea8f1-a4d4-4b15-ae66-77fcf02ec623","Type":"ContainerDied","Data":"17d28b0d0ced3e6aa44d3a402e16cb37cf4e532b8143c8c97269de7f4a2cdf7c"} Oct 31 16:04:03.427356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:03.427223 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17d28b0d0ced3e6aa44d3a402e16cb37cf4e532b8143c8c97269de7f4a2cdf7c" Oct 31 16:04:03.564090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:03.564051 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:05.294120 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.294087 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz"] Oct 31 16:04:05.294651 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.294319 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" containerID="cri-o://fbf18e8c199281845cecbaf21ee220c61e74dd3bfd4e49df433f5462d5d409df" gracePeriod=30 Oct 31 16:04:05.295198 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.295175 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:05.347543 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.347502 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2"] Oct 31 16:04:05.347807 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.347778 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" containerID="cri-o://99e5055bf12a74a16bb3f0ca718fbd8767f6c35720a250bd1e8e1ae810a6a682" gracePeriod=30 Oct 31 16:04:05.352158 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.352129 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh"] Oct 31 16:04:05.352401 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.352368 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" containerID="cri-o://8fb5bcbe22f079755b309de96787c0d3f3cafc52c26cccc84a0a86070a455348" gracePeriod=30 Oct 31 16:04:05.887771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.887707 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 16:04:05.984305 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:05.984278 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 16:04:06.000305 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.000277 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7049ebd-d545-4708-aec0-77058dd16d15-kube-api-access-vkxl2\" (UniqueName: \"kubernetes.io/projected/c7049ebd-d545-4708-aec0-77058dd16d15-kube-api-access-vkxl2\") pod \"c7049ebd-d545-4708-aec0-77058dd16d15\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " Oct 31 16:04:06.000564 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.000544 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7049ebd-d545-4708-aec0-77058dd16d15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7049ebd-d545-4708-aec0-77058dd16d15-plugins-config-volume\") pod \"c7049ebd-d545-4708-aec0-77058dd16d15\" (UID: \"c7049ebd-d545-4708-aec0-77058dd16d15\") " Oct 31 16:04:06.000881 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.000851 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7049ebd-d545-4708-aec0-77058dd16d15-plugins-config-volume" pod "c7049ebd-d545-4708-aec0-77058dd16d15" (UID: "c7049ebd-d545-4708-aec0-77058dd16d15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:06.001024 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.001007 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7049ebd-d545-4708-aec0-77058dd16d15-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:06.003054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.003025 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7049ebd-d545-4708-aec0-77058dd16d15-kube-api-access-vkxl2" pod "c7049ebd-d545-4708-aec0-77058dd16d15" (UID: "c7049ebd-d545-4708-aec0-77058dd16d15"). InnerVolumeSpecName "kube-api-access-vkxl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:06.101983 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.101951 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c23a96-8377-428b-942a-8bada9964aba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c23a96-8377-428b-942a-8bada9964aba-plugins-config-volume\") pod \"f9c23a96-8377-428b-942a-8bada9964aba\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " Oct 31 16:04:06.102173 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.102101 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c23a96-8377-428b-942a-8bada9964aba-kube-api-access-sqntc\" (UniqueName: \"kubernetes.io/projected/f9c23a96-8377-428b-942a-8bada9964aba-kube-api-access-sqntc\") pod \"f9c23a96-8377-428b-942a-8bada9964aba\" (UID: \"f9c23a96-8377-428b-942a-8bada9964aba\") " Oct 31 16:04:06.102341 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.102317 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c23a96-8377-428b-942a-8bada9964aba-plugins-config-volume" pod "f9c23a96-8377-428b-942a-8bada9964aba" (UID: "f9c23a96-8377-428b-942a-8bada9964aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:06.102429 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.102409 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c23a96-8377-428b-942a-8bada9964aba-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:06.102429 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.102423 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxl2\" (UniqueName: \"kubernetes.io/projected/c7049ebd-d545-4708-aec0-77058dd16d15-kube-api-access-vkxl2\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:06.104354 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.104333 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c23a96-8377-428b-942a-8bada9964aba-kube-api-access-sqntc" pod "f9c23a96-8377-428b-942a-8bada9964aba" (UID: "f9c23a96-8377-428b-942a-8bada9964aba"). InnerVolumeSpecName "kube-api-access-sqntc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:06.203926 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.203845 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqntc\" (UniqueName: \"kubernetes.io/projected/f9c23a96-8377-428b-942a-8bada9964aba-kube-api-access-sqntc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:06.450584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.450551 4865 generic.go:356] "Generic (PLEG): container finished" podID="f9c23a96-8377-428b-942a-8bada9964aba" containerID="fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0" exitCode=0 Oct 31 16:04:06.451051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.450600 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" Oct 31 16:04:06.451051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.450631 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" event={"ID":"f9c23a96-8377-428b-942a-8bada9964aba","Type":"ContainerDied","Data":"fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0"} Oct 31 16:04:06.451051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.450681 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx" event={"ID":"f9c23a96-8377-428b-942a-8bada9964aba","Type":"ContainerDied","Data":"104a6f5f6eb6732a8b8a8cf37bca16683664235e511162e311b35eb93cdaf763"} Oct 31 16:04:06.451051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.450702 4865 scope.go:122] "RemoveContainer" containerID="fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0" Oct 31 16:04:06.452303 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.452280 4865 generic.go:356] "Generic (PLEG): container finished" podID="c7049ebd-d545-4708-aec0-77058dd16d15" containerID="5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d" exitCode=0 Oct 31 16:04:06.452433 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.452340 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" Oct 31 16:04:06.452433 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.452380 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" event={"ID":"c7049ebd-d545-4708-aec0-77058dd16d15","Type":"ContainerDied","Data":"5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d"} Oct 31 16:04:06.452580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.452434 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn" event={"ID":"c7049ebd-d545-4708-aec0-77058dd16d15","Type":"ContainerDied","Data":"f78579e3c0efe4212b054a5cd78906a0bb0ca97eddced0b308eaa5dc9f1b3bad"} Oct 31 16:04:06.466431 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.466335 4865 scope.go:122] "RemoveContainer" containerID="fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0" Oct 31 16:04:06.466772 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:06.466749 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0\": container with ID starting with fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0 not found: ID does not exist" containerID="fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0" Oct 31 16:04:06.466886 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.466783 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0"} err="failed to get container status \"fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0\": rpc error: code = NotFound desc = could not find container \"fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0\": container with ID starting with fb3b1ce72df3cdf2ac520e4bc753ad6ebccea8696ff94b174135861983238ab0 not found: ID does not exist" Oct 31 16:04:06.466886 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.466807 4865 scope.go:122] "RemoveContainer" containerID="5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d" Oct 31 16:04:06.475950 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.475934 4865 scope.go:122] "RemoveContainer" containerID="5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d" Oct 31 16:04:06.476302 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:06.476276 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d\": container with ID starting with 5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d not found: ID does not exist" containerID="5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d" Oct 31 16:04:06.476378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.476313 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d"} err="failed to get container status \"5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d\": rpc error: code = NotFound desc = could not find container \"5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d\": container with ID starting with 5164673d2d880720bf8c816c558ebe11d709f6d8427b4478eef27dfae5d9f82d not found: ID does not exist" Oct 31 16:04:06.617927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.617895 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn"] Oct 31 16:04:06.622257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.622219 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-531c3c-8954cb8ff-xg9nn"] Oct 31 16:04:06.733424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.733340 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx"] Oct 31 16:04:06.739169 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:06.739144 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-531c3c-7ddf7c8f87-6l4hx"] Oct 31 16:04:07.273136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:07.273099 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-q54j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:04:07.273321 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:07.273166 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:04:07.498222 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:07.498194 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:07.633275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:07.633177 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:07.930760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:07.930676 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-n69gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:04:07.930760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:07.930747 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:04:08.496095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:08.496066 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7049ebd-d545-4708-aec0-77058dd16d15" path="/var/lib/kubelet/pods/c7049ebd-d545-4708-aec0-77058dd16d15/volumes" Oct 31 16:04:08.496501 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:08.496486 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c23a96-8377-428b-942a-8bada9964aba" path="/var/lib/kubelet/pods/f9c23a96-8377-428b-942a-8bada9964aba/volumes" Oct 31 16:04:09.048037 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.047998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:09.671724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.671690 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz"] Oct 31 16:04:09.671940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.671910 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" containerID="cri-o://11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685" gracePeriod=30 Oct 31 16:04:09.762632 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.762598 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:09.771229 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771202 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt"] Oct 31 16:04:09.771735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771721 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f9c23a96-8377-428b-942a-8bada9964aba" containerName="payload-pre-processing" Oct 31 16:04:09.771784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771737 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c23a96-8377-428b-942a-8bada9964aba" containerName="payload-pre-processing" Oct 31 16:04:09.771784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771775 4865 cpu_manager.go:414] "containerMap: removing container" podUID="53fea8f1-a4d4-4b15-ae66-77fcf02ec623" containerName="revoke-keys" Oct 31 16:04:09.771784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771780 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fea8f1-a4d4-4b15-ae66-77fcf02ec623" containerName="revoke-keys" Oct 31 16:04:09.771888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771812 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c7049ebd-d545-4708-aec0-77058dd16d15" containerName="payload-processing" Oct 31 16:04:09.771888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771818 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7049ebd-d545-4708-aec0-77058dd16d15" containerName="payload-processing" Oct 31 16:04:09.771958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771892 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fea8f1-a4d4-4b15-ae66-77fcf02ec623" containerName="revoke-keys" Oct 31 16:04:09.771958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771902 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7049ebd-d545-4708-aec0-77058dd16d15" containerName="payload-processing" Oct 31 16:04:09.771958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.771910 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c23a96-8377-428b-942a-8bada9964aba" containerName="payload-pre-processing" Oct 31 16:04:09.776708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.776691 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:09.866516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.851871 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt"] Oct 31 16:04:09.870781 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.870748 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:04:09.870952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.870845 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:04:09.885611 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.885582 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:04:09.885819 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.885640 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:04:09.954692 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.954603 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4dl\" (UniqueName: \"kubernetes.io/projected/a49ffa24-0879-4b9a-b393-59df15606b55-kube-api-access-bk4dl\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:09.954858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.954755 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:09.954903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:09.954873 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.055989 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.055952 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.056432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.056067 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.056432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.056143 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4dl\" (UniqueName: \"kubernetes.io/projected/a49ffa24-0879-4b9a-b393-59df15606b55-kube-api-access-bk4dl\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.056729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.056699 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.058733 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.058706 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.103102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.103066 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zmprh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:04:10.103305 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.103127 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:04:10.113608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.113584 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4dl\" (UniqueName: \"kubernetes.io/projected/a49ffa24-0879-4b9a-b393-59df15606b55-kube-api-access-bk4dl\") pod \"maas-api-revoke-sub-e0410ee3-mkhgt\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.149948 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.149916 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:10.390094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.390065 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:10.432738 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.432703 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:10.485951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.485420 4865 generic.go:356] "Generic (PLEG): container finished" podID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerID="fbf18e8c199281845cecbaf21ee220c61e74dd3bfd4e49df433f5462d5d409df" exitCode=0 Oct 31 16:04:10.485951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.485519 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" event={"ID":"7802fde4-f94f-49eb-be92-00fd1e5b5aec","Type":"ContainerDied","Data":"fbf18e8c199281845cecbaf21ee220c61e74dd3bfd4e49df433f5462d5d409df"} Oct 31 16:04:10.488815 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.488779 4865 generic.go:356] "Generic (PLEG): container finished" podID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerID="99e5055bf12a74a16bb3f0ca718fbd8767f6c35720a250bd1e8e1ae810a6a682" exitCode=0 Oct 31 16:04:10.491582 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.491553 4865 generic.go:356] "Generic (PLEG): container finished" podID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerID="8fb5bcbe22f079755b309de96787c0d3f3cafc52c26cccc84a0a86070a455348" exitCode=0 Oct 31 16:04:10.506494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.503094 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" event={"ID":"76fa31b0-d628-4102-9afb-2755cf2339cc","Type":"ContainerDied","Data":"99e5055bf12a74a16bb3f0ca718fbd8767f6c35720a250bd1e8e1ae810a6a682"} Oct 31 16:04:10.506494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.503157 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" event={"ID":"0eecba10-2f62-43a2-9ee0-f6ffb8973333","Type":"ContainerDied","Data":"8fb5bcbe22f079755b309de96787c0d3f3cafc52c26cccc84a0a86070a455348"} Oct 31 16:04:10.623173 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.623143 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 16:04:10.665295 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.665266 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 16:04:10.706387 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.706364 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 16:04:10.764797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.764768 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-envoy\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.764954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.764822 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-podinfo\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.764954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.764860 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-envoy\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.764954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.764896 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-certs\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.764954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.764945 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-socket\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.765216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765059 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-credential-socket\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.765216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765091 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-token\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-token\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.765216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765165 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istiod-ca-cert\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.765216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765203 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-kube-api-access-hvnsk\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-kube-api-access-hvnsk\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.765443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765240 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-podinfo\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.765443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765278 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-data\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.765443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765314 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-credential-socket\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.765443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765387 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-kube-api-access-5mkcj\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-kube-api-access-5mkcj\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.765443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765418 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fa31b0-d628-4102-9afb-2755cf2339cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fa31b0-d628-4102-9afb-2755cf2339cc-istiod-ca-cert\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.765769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765445 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-token\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-token\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.765769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765502 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-socket\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.765769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765559 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-certs\") pod \"76fa31b0-d628-4102-9afb-2755cf2339cc\" (UID: \"76fa31b0-d628-4102-9afb-2755cf2339cc\") " Oct 31 16:04:10.765769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.765596 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-data\") pod \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\" (UID: \"7802fde4-f94f-49eb-be92-00fd1e5b5aec\") " Oct 31 16:04:10.767327 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:04:10.766100 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49ffa24_0879_4b9a_b393_59df15606b55.slice/crio-8b382d9a55b1d3e6e9715838924312484634df18a837a2c01a3e62c32ca5bf3e WatchSource:0}: Error finding container 8b382d9a55b1d3e6e9715838924312484634df18a837a2c01a3e62c32ca5bf3e: Status 404 returned error can't find the container with id 8b382d9a55b1d3e6e9715838924312484634df18a837a2c01a3e62c32ca5bf3e Oct 31 16:04:10.767327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.766335 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-data" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.767327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.766430 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-credential-socket" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.767327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.766614 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-credential-socket" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.767327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.766697 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-data" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.767327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.766753 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fa31b0-d628-4102-9afb-2755cf2339cc-istiod-ca-cert" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:10.767327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.767086 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-socket" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.767849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.767372 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-certs" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.767849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.767750 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-envoy" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.768004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.767979 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-certs" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.768332 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.768309 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istiod-ca-cert" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:10.768572 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.768549 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-socket" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.769693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.769306 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-token" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:10.769693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.769532 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-envoy" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.769693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.769629 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-podinfo" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:04:10.770716 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.770690 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-kube-api-access-5mkcj" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "kube-api-access-5mkcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:10.770879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.770857 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-podinfo" pod "7802fde4-f94f-49eb-be92-00fd1e5b5aec" (UID: "7802fde4-f94f-49eb-be92-00fd1e5b5aec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:04:10.771261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.771192 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-token" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:10.771908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.771882 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-kube-api-access-hvnsk" pod "76fa31b0-d628-4102-9afb-2755cf2339cc" (UID: "76fa31b0-d628-4102-9afb-2755cf2339cc"). InnerVolumeSpecName "kube-api-access-hvnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:10.793773 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.793748 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt"] Oct 31 16:04:10.867344 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867270 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-podinfo\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867377 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-envoy\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867417 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-token\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-token\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867441 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-credential-socket\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867526 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-certs\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867550 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istiod-ca-cert\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867875 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867596 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-kube-api-access-9mnzf\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-kube-api-access-9mnzf\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867875 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867687 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-socket\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.867875 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867809 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-data\") pod \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\" (UID: \"0eecba10-2f62-43a2-9ee0-f6ffb8973333\") " Oct 31 16:04:10.868049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.867897 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-certs" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.868320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868151 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-credential-socket" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.868320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868134 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-socket" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.868320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868258 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istiod-ca-cert" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868669 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868688 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-data" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868695 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868755 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868771 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868781 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868790 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868800 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868809 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868818 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868827 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868835 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868844 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnsk\" (UniqueName: \"kubernetes.io/projected/76fa31b0-d628-4102-9afb-2755cf2339cc-kube-api-access-hvnsk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868854 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868863 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868871 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868880 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mkcj\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-kube-api-access-5mkcj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868888 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fa31b0-d628-4102-9afb-2755cf2339cc-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868896 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868906 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868915 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fa31b0-d628-4102-9afb-2755cf2339cc-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868923 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7802fde4-f94f-49eb-be92-00fd1e5b5aec-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.868949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.868931 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.870157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.870135 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-token" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:10.870226 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.870200 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-kube-api-access-9mnzf" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "kube-api-access-9mnzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:10.870409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.870396 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-envoy" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:10.871764 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.871736 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-podinfo" pod "0eecba10-2f62-43a2-9ee0-f6ffb8973333" (UID: "0eecba10-2f62-43a2-9ee0-f6ffb8973333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:04:10.970116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.970079 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.970116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.970110 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mnzf\" (UniqueName: \"kubernetes.io/projected/0eecba10-2f62-43a2-9ee0-f6ffb8973333-kube-api-access-9mnzf\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.970116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.970121 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.970385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.970130 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:10.970385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:10.970140 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eecba10-2f62-43a2-9ee0-f6ffb8973333-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:11.500704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.500671 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" Oct 31 16:04:11.500704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.500697 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2" event={"ID":"76fa31b0-d628-4102-9afb-2755cf2339cc","Type":"ContainerDied","Data":"383d1604a9b47cd7da4c32387f50cb74a0b1bc7a2a7971635db36db296284804"} Oct 31 16:04:11.501239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.500767 4865 scope.go:122] "RemoveContainer" containerID="99e5055bf12a74a16bb3f0ca718fbd8767f6c35720a250bd1e8e1ae810a6a682" Oct 31 16:04:11.508218 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.508192 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" Oct 31 16:04:11.508522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.508222 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh" event={"ID":"0eecba10-2f62-43a2-9ee0-f6ffb8973333","Type":"ContainerDied","Data":"819e407f8d93789377d1bd88bcaaa692945c8eb71eec47bb2e903ae0267bafa4"} Oct 31 16:04:11.510498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.510444 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" event={"ID":"7802fde4-f94f-49eb-be92-00fd1e5b5aec","Type":"ContainerDied","Data":"f5bd3a823c8819d6f09965674ff01db62edad4f87eab369298e24c9b6eb3ee5b"} Oct 31 16:04:11.510715 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.510655 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz" Oct 31 16:04:11.512678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.512644 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" event={"ID":"a49ffa24-0879-4b9a-b393-59df15606b55","Type":"ContainerStarted","Data":"10196b61a26f48d3a87eabab88beb7a161d6970fef0277564ae5b756e3fec4b9"} Oct 31 16:04:11.512795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.512691 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" event={"ID":"a49ffa24-0879-4b9a-b393-59df15606b55","Type":"ContainerStarted","Data":"8b382d9a55b1d3e6e9715838924312484634df18a837a2c01a3e62c32ca5bf3e"} Oct 31 16:04:11.514288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.514269 4865 scope.go:122] "RemoveContainer" containerID="8fb5bcbe22f079755b309de96787c0d3f3cafc52c26cccc84a0a86070a455348" Oct 31 16:04:11.525492 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.525457 4865 scope.go:122] "RemoveContainer" containerID="fbf18e8c199281845cecbaf21ee220c61e74dd3bfd4e49df433f5462d5d409df" Oct 31 16:04:11.669616 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.669581 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz"] Oct 31 16:04:11.720358 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.720329 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-n69gz"] Oct 31 16:04:11.773168 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.773092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:11.783261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.783231 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh"] Oct 31 16:04:11.786657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.786633 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zmprh"] Oct 31 16:04:11.809448 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.809415 4865 patch_prober.go:29] interesting pod/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:04:11.809613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.809497 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:04:11.897658 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.897624 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2"] Oct 31 16:04:11.901041 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:11.901020 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-q54j2"] Oct 31 16:04:12.012705 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:12.012650 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" podStartSLOduration=3.012634352 podStartE2EDuration="3.012634352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:04:12.011636749 +0000 UTC m=+4697.631188159" watchObservedRunningTime="2026-10-31 16:04:12.012634352 +0000 UTC m=+4697.632185740" Oct 31 16:04:12.499933 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:12.499900 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" path="/var/lib/kubelet/pods/0eecba10-2f62-43a2-9ee0-f6ffb8973333/volumes" Oct 31 16:04:12.500408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:12.500393 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" path="/var/lib/kubelet/pods/76fa31b0-d628-4102-9afb-2755cf2339cc/volumes" Oct 31 16:04:12.500834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:12.500821 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" path="/var/lib/kubelet/pods/7802fde4-f94f-49eb-be92-00fd1e5b5aec/volumes" Oct 31 16:04:12.522437 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:12.522401 4865 generic.go:356] "Generic (PLEG): container finished" podID="a49ffa24-0879-4b9a-b393-59df15606b55" containerID="10196b61a26f48d3a87eabab88beb7a161d6970fef0277564ae5b756e3fec4b9" exitCode=0 Oct 31 16:04:12.522636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:12.522450 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" event={"ID":"a49ffa24-0879-4b9a-b393-59df15606b55","Type":"ContainerDied","Data":"10196b61a26f48d3a87eabab88beb7a161d6970fef0277564ae5b756e3fec4b9"} Oct 31 16:04:13.270825 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.270785 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:13.690370 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.690343 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:13.801438 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.801353 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-xks-ca\") pod \"a49ffa24-0879-4b9a-b393-59df15606b55\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " Oct 31 16:04:13.801438 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.801425 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49ffa24-0879-4b9a-b393-59df15606b55-kube-api-access-bk4dl\" (UniqueName: \"kubernetes.io/projected/a49ffa24-0879-4b9a-b393-59df15606b55-kube-api-access-bk4dl\") pod \"a49ffa24-0879-4b9a-b393-59df15606b55\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " Oct 31 16:04:13.801659 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.801491 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-openshift-ca\") pod \"a49ffa24-0879-4b9a-b393-59df15606b55\" (UID: \"a49ffa24-0879-4b9a-b393-59df15606b55\") " Oct 31 16:04:13.801807 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.801781 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-openshift-ca" pod "a49ffa24-0879-4b9a-b393-59df15606b55" (UID: "a49ffa24-0879-4b9a-b393-59df15606b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:13.801934 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.801920 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:13.803689 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.803662 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-xks-ca" pod "a49ffa24-0879-4b9a-b393-59df15606b55" (UID: "a49ffa24-0879-4b9a-b393-59df15606b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:04:13.803803 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.803688 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49ffa24-0879-4b9a-b393-59df15606b55-kube-api-access-bk4dl" pod "a49ffa24-0879-4b9a-b393-59df15606b55" (UID: "a49ffa24-0879-4b9a-b393-59df15606b55"). InnerVolumeSpecName "kube-api-access-bk4dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:13.903268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.903227 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49ffa24-0879-4b9a-b393-59df15606b55-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:13.903268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:13.903266 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk4dl\" (UniqueName: \"kubernetes.io/projected/a49ffa24-0879-4b9a-b393-59df15606b55-kube-api-access-bk4dl\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:14.538631 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:14.538602 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" Oct 31 16:04:14.538802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:14.538632 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt" event={"ID":"a49ffa24-0879-4b9a-b393-59df15606b55","Type":"ContainerDied","Data":"8b382d9a55b1d3e6e9715838924312484634df18a837a2c01a3e62c32ca5bf3e"} Oct 31 16:04:14.538802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:14.538674 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b382d9a55b1d3e6e9715838924312484634df18a837a2c01a3e62c32ca5bf3e" Oct 31 16:04:14.898058 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:14.898024 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:14.965215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:14.965190 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 16:04:15.119676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.119562 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924114e3-1c2c-432e-8492-b7f6811359ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924114e3-1c2c-432e-8492-b7f6811359ea-istio-podinfo\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.119676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.119660 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-socket\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.119951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.119750 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-kube-api-access-5st22\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-kube-api-access-5st22\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.119951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.119860 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-credential-socket\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.120082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.119982 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-certs\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.120082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.120004 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-socket" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:15.120082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.120017 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-istio-token\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-istio-token\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.120339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.120156 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-data\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.120339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.120227 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924114e3-1c2c-432e-8492-b7f6811359ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924114e3-1c2c-432e-8492-b7f6811359ea-istiod-ca-cert\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.120339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.120240 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-certs" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:15.120557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.120434 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-credential-socket" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:15.120630 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.120570 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-data" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:15.121038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.121011 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924114e3-1c2c-432e-8492-b7f6811359ea-istiod-ca-cert" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:15.121312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.121091 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-envoy\") pod \"924114e3-1c2c-432e-8492-b7f6811359ea\" (UID: \"924114e3-1c2c-432e-8492-b7f6811359ea\") " Oct 31 16:04:15.121631 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.121607 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.121742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.121634 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.121742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.121649 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924114e3-1c2c-432e-8492-b7f6811359ea-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.121742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.121665 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.121742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.121679 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.122642 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.122619 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-kube-api-access-5st22" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "kube-api-access-5st22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:15.122820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.122794 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-istio-token" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:15.123300 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.123281 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924114e3-1c2c-432e-8492-b7f6811359ea-istio-podinfo" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:04:15.123608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.123591 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-envoy" pod "924114e3-1c2c-432e-8492-b7f6811359ea" (UID: "924114e3-1c2c-432e-8492-b7f6811359ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:15.222609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.222562 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.222609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.222600 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924114e3-1c2c-432e-8492-b7f6811359ea-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.222609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.222614 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924114e3-1c2c-432e-8492-b7f6811359ea-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.222865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.222627 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5st22\" (UniqueName: \"kubernetes.io/projected/924114e3-1c2c-432e-8492-b7f6811359ea-kube-api-access-5st22\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:15.547574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.547541 4865 generic.go:356] "Generic (PLEG): container finished" podID="924114e3-1c2c-432e-8492-b7f6811359ea" containerID="11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685" exitCode=0 Oct 31 16:04:15.547794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.547600 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" Oct 31 16:04:15.547794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.547622 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" event={"ID":"924114e3-1c2c-432e-8492-b7f6811359ea","Type":"ContainerDied","Data":"11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685"} Oct 31 16:04:15.547794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.547679 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz" event={"ID":"924114e3-1c2c-432e-8492-b7f6811359ea","Type":"ContainerDied","Data":"9ceb77ab324e5f5ba9f3281eb8c83fea4f67347bdbb817be0580139505494769"} Oct 31 16:04:15.547794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.547702 4865 scope.go:122] "RemoveContainer" containerID="11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685" Oct 31 16:04:15.558596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.558577 4865 scope.go:122] "RemoveContainer" containerID="b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605" Oct 31 16:04:15.568068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.568048 4865 scope.go:122] "RemoveContainer" containerID="11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685" Oct 31 16:04:15.568365 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:15.568340 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685\": container with ID starting with 11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685 not found: ID does not exist" containerID="11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685" Oct 31 16:04:15.568507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.568369 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685"} err="failed to get container status \"11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685\": rpc error: code = NotFound desc = could not find container \"11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685\": container with ID starting with 11b586a54bb4629bfa7a0c304aac237baa16edac9fc3109707333258bb935685 not found: ID does not exist" Oct 31 16:04:15.568507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.568386 4865 scope.go:122] "RemoveContainer" containerID="b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605" Oct 31 16:04:15.568662 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:15.568644 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605\": container with ID starting with b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605 not found: ID does not exist" containerID="b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605" Oct 31 16:04:15.568701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.568669 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605"} err="failed to get container status \"b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605\": rpc error: code = NotFound desc = could not find container \"b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605\": container with ID starting with b0332d44bb6195ca2bf71543e288908ab185ff3f57b47bf318351638f0639605 not found: ID does not exist" Oct 31 16:04:15.716439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.716349 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz"] Oct 31 16:04:15.767232 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:15.767202 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-531c3c-openshift-default-7657dc6df7-tnnbz"] Oct 31 16:04:16.235685 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:16.235635 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:16.496159 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:16.496084 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" path="/var/lib/kubelet/pods/924114e3-1c2c-432e-8492-b7f6811359ea/volumes" Oct 31 16:04:16.855044 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:16.854948 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh"] Oct 31 16:04:16.964286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:16.964254 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv"] Oct 31 16:04:17.207717 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:17.207679 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:17.327144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:17.327114 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:17.407360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:17.407326 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:04:17.490820 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:17.490730 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:04:20.030699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:20.030661 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:20.313445 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:20.313352 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:20.447195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:20.447152 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:21.855724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:21.855675 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" podUID="e424c501-ac9c-400a-90df-1062892d7e7f" containerName="payload-processing" containerID="cri-o://7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2" gracePeriod=25 Oct 31 16:04:21.965023 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:21.964979 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" podUID="a11b4071-920e-4101-99c4-f4809253ae82" containerName="payload-pre-processing" containerID="cri-o://ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e" gracePeriod=25 Oct 31 16:04:22.044707 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:22.044671 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:23.147045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:23.147008 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:24.696606 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:24.696576 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:26.143626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:26.143591 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" podUID="a11b4071-920e-4101-99c4-f4809253ae82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:26.144162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:26.143639 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" podUID="e424c501-ac9c-400a-90df-1062892d7e7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:29.384049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:29.384012 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:34.009798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:34.009760 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:36.144277 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:36.144078 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" podUID="e424c501-ac9c-400a-90df-1062892d7e7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:36.144840 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:36.144413 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" podUID="a11b4071-920e-4101-99c4-f4809253ae82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:38.285087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:38.285055 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:38.420057 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:38.419998 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:38.775450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:38.775419 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:42.125330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.125268 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 16:04:42.217935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.217913 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 16:04:42.218133 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.218114 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e424c501-ac9c-400a-90df-1062892d7e7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e424c501-ac9c-400a-90df-1062892d7e7f-plugins-config-volume\") pod \"e424c501-ac9c-400a-90df-1062892d7e7f\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " Oct 31 16:04:42.218256 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.218233 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e424c501-ac9c-400a-90df-1062892d7e7f-kube-api-access-wsffv\" (UniqueName: \"kubernetes.io/projected/e424c501-ac9c-400a-90df-1062892d7e7f-kube-api-access-wsffv\") pod \"e424c501-ac9c-400a-90df-1062892d7e7f\" (UID: \"e424c501-ac9c-400a-90df-1062892d7e7f\") " Oct 31 16:04:42.218465 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.218447 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e424c501-ac9c-400a-90df-1062892d7e7f-plugins-config-volume" pod "e424c501-ac9c-400a-90df-1062892d7e7f" (UID: "e424c501-ac9c-400a-90df-1062892d7e7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:42.218814 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.218793 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e424c501-ac9c-400a-90df-1062892d7e7f-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:42.220399 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.220382 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e424c501-ac9c-400a-90df-1062892d7e7f-kube-api-access-wsffv" pod "e424c501-ac9c-400a-90df-1062892d7e7f" (UID: "e424c501-ac9c-400a-90df-1062892d7e7f"). InnerVolumeSpecName "kube-api-access-wsffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:42.319675 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.319641 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11b4071-920e-4101-99c4-f4809253ae82-kube-api-access-ql8gw\" (UniqueName: \"kubernetes.io/projected/a11b4071-920e-4101-99c4-f4809253ae82-kube-api-access-ql8gw\") pod \"a11b4071-920e-4101-99c4-f4809253ae82\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " Oct 31 16:04:42.319877 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.319802 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11b4071-920e-4101-99c4-f4809253ae82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11b4071-920e-4101-99c4-f4809253ae82-plugins-config-volume\") pod \"a11b4071-920e-4101-99c4-f4809253ae82\" (UID: \"a11b4071-920e-4101-99c4-f4809253ae82\") " Oct 31 16:04:42.320140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.320122 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsffv\" (UniqueName: \"kubernetes.io/projected/e424c501-ac9c-400a-90df-1062892d7e7f-kube-api-access-wsffv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:42.320204 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.320153 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11b4071-920e-4101-99c4-f4809253ae82-plugins-config-volume" pod "a11b4071-920e-4101-99c4-f4809253ae82" (UID: "a11b4071-920e-4101-99c4-f4809253ae82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:42.321988 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.321964 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11b4071-920e-4101-99c4-f4809253ae82-kube-api-access-ql8gw" pod "a11b4071-920e-4101-99c4-f4809253ae82" (UID: "a11b4071-920e-4101-99c4-f4809253ae82"). InnerVolumeSpecName "kube-api-access-ql8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:42.421448 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.421362 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8gw\" (UniqueName: \"kubernetes.io/projected/a11b4071-920e-4101-99c4-f4809253ae82-kube-api-access-ql8gw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:42.421448 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.421394 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11b4071-920e-4101-99c4-f4809253ae82-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:42.714355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.714322 4865 generic.go:356] "Generic (PLEG): container finished" podID="e424c501-ac9c-400a-90df-1062892d7e7f" containerID="7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2" exitCode=0 Oct 31 16:04:42.714593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.714384 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" Oct 31 16:04:42.714593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.714402 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" event={"ID":"e424c501-ac9c-400a-90df-1062892d7e7f","Type":"ContainerDied","Data":"7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2"} Oct 31 16:04:42.714593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.714460 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh" event={"ID":"e424c501-ac9c-400a-90df-1062892d7e7f","Type":"ContainerDied","Data":"e9701b34be885faa3f53f7b6756cd4e779ea49f9735a3be4ed3923afa6c482a5"} Oct 31 16:04:42.714593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.714506 4865 scope.go:122] "RemoveContainer" containerID="7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2" Oct 31 16:04:42.716152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.716126 4865 generic.go:356] "Generic (PLEG): container finished" podID="a11b4071-920e-4101-99c4-f4809253ae82" containerID="ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e" exitCode=0 Oct 31 16:04:42.716282 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.716195 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" Oct 31 16:04:42.716282 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.716219 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" event={"ID":"a11b4071-920e-4101-99c4-f4809253ae82","Type":"ContainerDied","Data":"ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e"} Oct 31 16:04:42.716282 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.716257 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv" event={"ID":"a11b4071-920e-4101-99c4-f4809253ae82","Type":"ContainerDied","Data":"8a4c3dde8e5614eb74700cb11157776d9eab47abce93f3e6d118c4d1b3ba13d5"} Oct 31 16:04:42.724827 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.724813 4865 scope.go:122] "RemoveContainer" containerID="7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2" Oct 31 16:04:42.725043 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:42.725027 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2\": container with ID starting with 7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2 not found: ID does not exist" containerID="7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2" Oct 31 16:04:42.725093 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.725051 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2"} err="failed to get container status \"7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2\": rpc error: code = NotFound desc = could not find container \"7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2\": container with ID starting with 7c74eeb5692c148fc6a1967b192ba1fcdb7b8ed0b614d03173f6ea7b26cfefa2 not found: ID does not exist" Oct 31 16:04:42.725093 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.725065 4865 scope.go:122] "RemoveContainer" containerID="ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e" Oct 31 16:04:42.734290 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.734272 4865 scope.go:122] "RemoveContainer" containerID="ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e" Oct 31 16:04:42.734555 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:42.734535 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e\": container with ID starting with ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e not found: ID does not exist" containerID="ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e" Oct 31 16:04:42.734611 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.734560 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e"} err="failed to get container status \"ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e\": rpc error: code = NotFound desc = could not find container \"ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e\": container with ID starting with ba2b9345cce9e204859cd5e1c6980ea48f9d22420d9c530dcd9b745e45283c7e not found: ID does not exist" Oct 31 16:04:42.880316 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.880286 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv"] Oct 31 16:04:42.931425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.931391 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-55b3fb-7bf96b6879-p5czv"] Oct 31 16:04:42.992525 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.992413 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh"] Oct 31 16:04:42.995983 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:42.995946 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-55b3fb-7d86c48f95-7fzhh"] Oct 31 16:04:43.357061 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:43.356972 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:43.481016 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:43.480981 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:43.813771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:43.813737 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:44.496462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:44.496421 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11b4071-920e-4101-99c4-f4809253ae82" path="/var/lib/kubelet/pods/a11b4071-920e-4101-99c4-f4809253ae82/volumes" Oct 31 16:04:44.497009 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:44.496989 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e424c501-ac9c-400a-90df-1062892d7e7f" path="/var/lib/kubelet/pods/e424c501-ac9c-400a-90df-1062892d7e7f/volumes" Oct 31 16:04:47.487398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:47.487359 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:47.490507 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:04:47.490461 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:04:49.174551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:49.174510 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:49.186108 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:49.186074 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9"] Oct 31 16:04:49.293656 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:49.293598 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c"] Oct 31 16:04:49.537170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:49.537133 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:49.655819 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:49.655785 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:49.789183 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:49.789092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:53.419188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:53.419145 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:53.556660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:53.556609 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:54.073369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:54.073334 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:54.186836 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:54.186688 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" podUID="5204b4e7-05da-4091-8727-83cb64939951" containerName="payload-processing" containerID="cri-o://df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079" gracePeriod=25 Oct 31 16:04:54.294940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:54.294897 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" podUID="ebf94031-4801-4661-a79a-a80fed8625d3" containerName="payload-pre-processing" containerID="cri-o://720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852" gracePeriod=25 Oct 31 16:04:54.496271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:54.496240 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:04:54.931225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:54.931131 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" podUID="5204b4e7-05da-4091-8727-83cb64939951" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:54.931399 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:54.931136 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" podUID="ebf94031-4801-4661-a79a-a80fed8625d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:04:54.967455 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:54.967424 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:55.489157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:55.489123 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw"] Oct 31 16:04:55.489370 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:55.489320 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" containerID="cri-o://fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8" gracePeriod=30 Oct 31 16:04:55.516584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:55.516558 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:55.945630 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:55.945554 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:56.082296 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.082224 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:56.558762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.558723 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55"] Oct 31 16:04:56.559255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559239 4865 cpu_manager.go:414] "containerMap: removing container" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" Oct 31 16:04:56.559304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559257 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" Oct 31 16:04:56.559304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559273 4865 cpu_manager.go:414] "containerMap: removing container" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" Oct 31 16:04:56.559304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559279 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" Oct 31 16:04:56.559304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559286 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a11b4071-920e-4101-99c4-f4809253ae82" containerName="payload-pre-processing" Oct 31 16:04:56.559304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559291 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11b4071-920e-4101-99c4-f4809253ae82" containerName="payload-pre-processing" Oct 31 16:04:56.559304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559300 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a49ffa24-0879-4b9a-b393-59df15606b55" containerName="revoke-keys" Oct 31 16:04:56.559304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559304 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49ffa24-0879-4b9a-b393-59df15606b55" containerName="revoke-keys" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559326 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e424c501-ac9c-400a-90df-1062892d7e7f" containerName="payload-processing" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559331 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e424c501-ac9c-400a-90df-1062892d7e7f" containerName="payload-processing" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559341 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559346 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559362 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559367 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559437 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559448 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eecba10-2f62-43a2-9ee0-f6ffb8973333" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559457 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fa31b0-d628-4102-9afb-2755cf2339cc" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559477 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559483 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11b4071-920e-4101-99c4-f4809253ae82" containerName="payload-pre-processing" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559489 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49ffa24-0879-4b9a-b393-59df15606b55" containerName="revoke-keys" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559498 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e424c501-ac9c-400a-90df-1062892d7e7f" containerName="payload-processing" Oct 31 16:04:56.559592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.559507 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7802fde4-f94f-49eb-be92-00fd1e5b5aec" containerName="istio-proxy" Oct 31 16:04:56.565757 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.565726 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.617745 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.617716 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55"] Oct 31 16:04:56.665413 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.665388 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:04:56.665608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.665423 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:04:56.665608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.665439 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:04:56.670012 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.669994 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:04:56.686749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.686727 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.686892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.686764 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsgk\" (UniqueName: \"kubernetes.io/projected/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-kube-api-access-jzsgk\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.686892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.686796 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.787408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.787377 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.787616 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.787425 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsgk\" (UniqueName: \"kubernetes.io/projected/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-kube-api-access-jzsgk\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.787616 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.787483 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.788228 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.788158 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.789958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.789938 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.848239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.848176 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsgk\" (UniqueName: \"kubernetes.io/projected/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-kube-api-access-jzsgk\") pod \"maas-api-revoke-sub-fdabb34a-h8v55\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:56.882525 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:56.882501 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:04:57.254412 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:57.254386 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55"] Oct 31 16:04:57.254764 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:04:57.254738 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d35429_f86b_4a5c_9ec6_6b48c5c843bd.slice/crio-214db65e0ae12049d9daad9dc444956be0582d1f3da47e0edc27f668fb42c29b WatchSource:0}: Error finding container 214db65e0ae12049d9daad9dc444956be0582d1f3da47e0edc27f668fb42c29b: Status 404 returned error can't find the container with id 214db65e0ae12049d9daad9dc444956be0582d1f3da47e0edc27f668fb42c29b Oct 31 16:04:57.824521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:57.824431 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" event={"ID":"79d35429-f86b-4a5c-9ec6-6b48c5c843bd","Type":"ContainerStarted","Data":"067ba4945a3fce56b21b84b552db7777da82212ea0343e58abe42ef0c11ffb5e"} Oct 31 16:04:57.824521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:57.824503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" event={"ID":"79d35429-f86b-4a5c-9ec6-6b48c5c843bd","Type":"ContainerStarted","Data":"214db65e0ae12049d9daad9dc444956be0582d1f3da47e0edc27f668fb42c29b"} Oct 31 16:04:57.939854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:57.939799 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" podStartSLOduration=1.939785388 podStartE2EDuration="1.939785388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:04:57.93825875 +0000 UTC m=+4743.557810146" watchObservedRunningTime="2026-10-31 16:04:57.939785388 +0000 UTC m=+4743.559336776" Oct 31 16:04:58.831914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:58.831877 4865 generic.go:356] "Generic (PLEG): container finished" podID="79d35429-f86b-4a5c-9ec6-6b48c5c843bd" containerID="067ba4945a3fce56b21b84b552db7777da82212ea0343e58abe42ef0c11ffb5e" exitCode=0 Oct 31 16:04:58.832342 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:58.831967 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" event={"ID":"79d35429-f86b-4a5c-9ec6-6b48c5c843bd","Type":"ContainerDied","Data":"067ba4945a3fce56b21b84b552db7777da82212ea0343e58abe42ef0c11ffb5e"} Oct 31 16:04:59.151704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:59.151611 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:59.979158 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:04:59.979133 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:05:00.019239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.019210 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-xks-ca\") pod \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " Oct 31 16:05:00.019426 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.019255 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-kube-api-access-jzsgk\" (UniqueName: \"kubernetes.io/projected/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-kube-api-access-jzsgk\") pod \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " Oct 31 16:05:00.019426 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.019310 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-openshift-ca\") pod \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\" (UID: \"79d35429-f86b-4a5c-9ec6-6b48c5c843bd\") " Oct 31 16:05:00.019711 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.019687 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-openshift-ca" pod "79d35429-f86b-4a5c-9ec6-6b48c5c843bd" (UID: "79d35429-f86b-4a5c-9ec6-6b48c5c843bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:00.019891 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.019870 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.021423 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.021396 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-xks-ca" pod "79d35429-f86b-4a5c-9ec6-6b48c5c843bd" (UID: "79d35429-f86b-4a5c-9ec6-6b48c5c843bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:00.021564 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.021536 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-kube-api-access-jzsgk" pod "79d35429-f86b-4a5c-9ec6-6b48c5c843bd" (UID: "79d35429-f86b-4a5c-9ec6-6b48c5c843bd"). InnerVolumeSpecName "kube-api-access-jzsgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:00.121381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.121299 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.121381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.121339 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzsgk\" (UniqueName: \"kubernetes.io/projected/79d35429-f86b-4a5c-9ec6-6b48c5c843bd-kube-api-access-jzsgk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.741380 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.741353 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 16:05:00.829462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829380 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-socket\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.829647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829462 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-istio-token\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-istio-token\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.829647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829533 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf89f0e4-3499-4802-9a05-72062f42def8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89f0e4-3499-4802-9a05-72062f42def8-istio-podinfo\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.829647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829570 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-credential-socket\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.829647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829633 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-kube-api-access-ctg28\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-kube-api-access-ctg28\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.829851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829662 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-envoy\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.829851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829707 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-certs\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.829851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829706 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-socket" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:00.829851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829766 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf89f0e4-3499-4802-9a05-72062f42def8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89f0e4-3499-4802-9a05-72062f42def8-istiod-ca-cert\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.830080 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829849 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-data\") pod \"cf89f0e4-3499-4802-9a05-72062f42def8\" (UID: \"cf89f0e4-3499-4802-9a05-72062f42def8\") " Oct 31 16:05:00.830080 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829925 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-credential-socket" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:00.830080 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.829971 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-certs" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:00.830355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.830122 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf89f0e4-3499-4802-9a05-72062f42def8-istiod-ca-cert" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:00.830355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.830188 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-data" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:00.830661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.830373 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.830661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.830393 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.830661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.830402 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89f0e4-3499-4802-9a05-72062f42def8-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.830661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.830410 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.830661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.830418 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.832022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.831990 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf89f0e4-3499-4802-9a05-72062f42def8-istio-podinfo" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:05:00.832022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.832005 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-kube-api-access-ctg28" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "kube-api-access-ctg28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:00.832175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.832025 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-envoy" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:00.832241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.832226 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-istio-token" pod "cf89f0e4-3499-4802-9a05-72062f42def8" (UID: "cf89f0e4-3499-4802-9a05-72062f42def8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:00.845981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.845959 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" Oct 31 16:05:00.845981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.845973 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55" event={"ID":"79d35429-f86b-4a5c-9ec6-6b48c5c843bd","Type":"ContainerDied","Data":"214db65e0ae12049d9daad9dc444956be0582d1f3da47e0edc27f668fb42c29b"} Oct 31 16:05:00.846131 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.846018 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="214db65e0ae12049d9daad9dc444956be0582d1f3da47e0edc27f668fb42c29b" Oct 31 16:05:00.847798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.847778 4865 generic.go:356] "Generic (PLEG): container finished" podID="cf89f0e4-3499-4802-9a05-72062f42def8" containerID="fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8" exitCode=0 Oct 31 16:05:00.847915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.847824 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" Oct 31 16:05:00.847915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.847843 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" event={"ID":"cf89f0e4-3499-4802-9a05-72062f42def8","Type":"ContainerDied","Data":"fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8"} Oct 31 16:05:00.848030 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.847932 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw" event={"ID":"cf89f0e4-3499-4802-9a05-72062f42def8","Type":"ContainerDied","Data":"5a60d974bdc5da0db40657c00b59a261e209a74709b5521adbfe762723dbe9bc"} Oct 31 16:05:00.848030 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.847971 4865 scope.go:122] "RemoveContainer" containerID="fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8" Oct 31 16:05:00.858261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.858237 4865 scope.go:122] "RemoveContainer" containerID="2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754" Oct 31 16:05:00.867742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.867726 4865 scope.go:122] "RemoveContainer" containerID="fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8" Oct 31 16:05:00.867982 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:00.867963 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8\": container with ID starting with fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8 not found: ID does not exist" containerID="fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8" Oct 31 16:05:00.868060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.867993 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8"} err="failed to get container status \"fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8\": rpc error: code = NotFound desc = could not find container \"fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8\": container with ID starting with fdaf4df87ab67c74fb72c8271b15f0653c6b6ca506df4840d265cb1016e712c8 not found: ID does not exist" Oct 31 16:05:00.868060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.868021 4865 scope.go:122] "RemoveContainer" containerID="2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754" Oct 31 16:05:00.868281 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:00.868262 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754\": container with ID starting with 2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754 not found: ID does not exist" containerID="2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754" Oct 31 16:05:00.868320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.868286 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754"} err="failed to get container status \"2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754\": rpc error: code = NotFound desc = could not find container \"2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754\": container with ID starting with 2d76f755fc0f122e3f148ab81c336e07e281d9e7b6cea7fbf5bc24b11b2ae754 not found: ID does not exist" Oct 31 16:05:00.932210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.932182 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.932210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.932208 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89f0e4-3499-4802-9a05-72062f42def8-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.932385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.932219 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctg28\" (UniqueName: \"kubernetes.io/projected/cf89f0e4-3499-4802-9a05-72062f42def8-kube-api-access-ctg28\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.932385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.932230 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89f0e4-3499-4802-9a05-72062f42def8-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:00.967245 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:00.967204 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk"] Oct 31 16:05:01.030963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:01.030925 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:01.127138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:01.127049 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw"] Oct 31 16:05:01.130772 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:01.130747 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-55b3fb-openshift-default-bccc556fc-5s2pw"] Oct 31 16:05:02.497026 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:02.496991 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" path="/var/lib/kubelet/pods/cf89f0e4-3499-4802-9a05-72062f42def8/volumes" Oct 31 16:05:03.902569 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:03.902540 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:04.931093 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:04.931052 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" podUID="5204b4e7-05da-4091-8727-83cb64939951" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:04.931548 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:04.931052 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" podUID="ebf94031-4801-4661-a79a-a80fed8625d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:07.160490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:07.160431 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:08.601378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:08.601340 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:12.148090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.148060 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:12.282776 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.282747 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf"] Oct 31 16:05:12.283221 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283209 4865 cpu_manager.go:414] "containerMap: removing container" podUID="79d35429-f86b-4a5c-9ec6-6b48c5c843bd" containerName="revoke-keys" Oct 31 16:05:12.283274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283223 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d35429-f86b-4a5c-9ec6-6b48c5c843bd" containerName="revoke-keys" Oct 31 16:05:12.283274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283244 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" Oct 31 16:05:12.283274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283249 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" Oct 31 16:05:12.283274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283265 4865 cpu_manager.go:414] "containerMap: removing container" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" Oct 31 16:05:12.283274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283270 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924114e3-1c2c-432e-8492-b7f6811359ea" containerName="istio-proxy" Oct 31 16:05:12.283443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283355 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" Oct 31 16:05:12.283443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.283366 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d35429-f86b-4a5c-9ec6-6b48c5c843bd" containerName="revoke-keys" Oct 31 16:05:12.286334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.286317 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.347194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.347168 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf"] Oct 31 16:05:12.395749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.395717 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:05:12.395913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.395747 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:05:12.395913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.395792 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:05:12.396006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.395948 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:05:12.414538 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.414459 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz"] Oct 31 16:05:12.453190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.453166 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.453304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.453196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tsq\" (UniqueName: \"kubernetes.io/projected/c02b3b7a-88d8-4325-a130-0965a02c1109-kube-api-access-d6tsq\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.453358 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.453347 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.497223 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.497201 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2"] Oct 31 16:05:12.554416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.554386 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.554416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.554415 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tsq\" (UniqueName: \"kubernetes.io/projected/c02b3b7a-88d8-4325-a130-0965a02c1109-kube-api-access-d6tsq\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.554627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.554539 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.555031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.555011 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.556996 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.556974 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.612743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.612720 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tsq\" (UniqueName: \"kubernetes.io/projected/c02b3b7a-88d8-4325-a130-0965a02c1109-kube-api-access-d6tsq\") pod \"maas-api-revoke-sub-d06a99e9-vpkgf\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:12.750973 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.750932 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:12.876056 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.876021 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:12.899127 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:12.899104 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:13.281966 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:13.281943 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf"] Oct 31 16:05:13.283537 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:05:13.283505 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02b3b7a_88d8_4325_a130_0965a02c1109.slice/crio-94d5bcec8a0887486ae4ac07e1ac95d75b208fb4724b0bb4326abaa7a948ef17 WatchSource:0}: Error finding container 94d5bcec8a0887486ae4ac07e1ac95d75b208fb4724b0bb4326abaa7a948ef17: Status 404 returned error can't find the container with id 94d5bcec8a0887486ae4ac07e1ac95d75b208fb4724b0bb4326abaa7a948ef17 Oct 31 16:05:13.299821 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:13.299796 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:13.424985 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:13.424956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:13.614824 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:13.614799 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:13.931927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:13.931829 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" event={"ID":"c02b3b7a-88d8-4325-a130-0965a02c1109","Type":"ContainerStarted","Data":"b2c675b6dde40b6727a4e7672232dd93b92b84ece717761412a6285c26906064"} Oct 31 16:05:13.931927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:13.931893 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" event={"ID":"c02b3b7a-88d8-4325-a130-0965a02c1109","Type":"ContainerStarted","Data":"94d5bcec8a0887486ae4ac07e1ac95d75b208fb4724b0bb4326abaa7a948ef17"} Oct 31 16:05:14.049988 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.049936 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" podStartSLOduration=2.049921907 podStartE2EDuration="2.049921907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:05:14.047195611 +0000 UTC m=+4759.666747023" watchObservedRunningTime="2026-10-31 16:05:14.049921907 +0000 UTC m=+4759.669473317" Oct 31 16:05:14.458998 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.458972 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 16:05:14.541383 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.541359 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 16:05:14.575424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.575398 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5204b4e7-05da-4091-8727-83cb64939951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5204b4e7-05da-4091-8727-83cb64939951-plugins-config-volume\") pod \"5204b4e7-05da-4091-8727-83cb64939951\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " Oct 31 16:05:14.575562 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.575543 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5204b4e7-05da-4091-8727-83cb64939951-kube-api-access-lqj6s\" (UniqueName: \"kubernetes.io/projected/5204b4e7-05da-4091-8727-83cb64939951-kube-api-access-lqj6s\") pod \"5204b4e7-05da-4091-8727-83cb64939951\" (UID: \"5204b4e7-05da-4091-8727-83cb64939951\") " Oct 31 16:05:14.576273 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.576199 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5204b4e7-05da-4091-8727-83cb64939951-plugins-config-volume" pod "5204b4e7-05da-4091-8727-83cb64939951" (UID: "5204b4e7-05da-4091-8727-83cb64939951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:14.578421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.578395 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5204b4e7-05da-4091-8727-83cb64939951-kube-api-access-lqj6s" pod "5204b4e7-05da-4091-8727-83cb64939951" (UID: "5204b4e7-05da-4091-8727-83cb64939951"). InnerVolumeSpecName "kube-api-access-lqj6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:14.677386 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.677361 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf94031-4801-4661-a79a-a80fed8625d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf94031-4801-4661-a79a-a80fed8625d3-plugins-config-volume\") pod \"ebf94031-4801-4661-a79a-a80fed8625d3\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " Oct 31 16:05:14.677577 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.677523 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf94031-4801-4661-a79a-a80fed8625d3-kube-api-access-jrpvh\" (UniqueName: \"kubernetes.io/projected/ebf94031-4801-4661-a79a-a80fed8625d3-kube-api-access-jrpvh\") pod \"ebf94031-4801-4661-a79a-a80fed8625d3\" (UID: \"ebf94031-4801-4661-a79a-a80fed8625d3\") " Oct 31 16:05:14.677764 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.677742 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf94031-4801-4661-a79a-a80fed8625d3-plugins-config-volume" pod "ebf94031-4801-4661-a79a-a80fed8625d3" (UID: "ebf94031-4801-4661-a79a-a80fed8625d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:14.677879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.677864 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5204b4e7-05da-4091-8727-83cb64939951-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:14.677929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.677882 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqj6s\" (UniqueName: \"kubernetes.io/projected/5204b4e7-05da-4091-8727-83cb64939951-kube-api-access-lqj6s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:14.677929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.677893 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf94031-4801-4661-a79a-a80fed8625d3-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:14.679722 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.679699 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf94031-4801-4661-a79a-a80fed8625d3-kube-api-access-jrpvh" pod "ebf94031-4801-4661-a79a-a80fed8625d3" (UID: "ebf94031-4801-4661-a79a-a80fed8625d3"). InnerVolumeSpecName "kube-api-access-jrpvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:14.778612 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.778528 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrpvh\" (UniqueName: \"kubernetes.io/projected/ebf94031-4801-4661-a79a-a80fed8625d3-kube-api-access-jrpvh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:14.813969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.813940 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:05:14.938883 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.938848 4865 generic.go:356] "Generic (PLEG): container finished" podID="ebf94031-4801-4661-a79a-a80fed8625d3" containerID="720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852" exitCode=0 Oct 31 16:05:14.939047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.938894 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" Oct 31 16:05:14.939047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.938933 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" event={"ID":"ebf94031-4801-4661-a79a-a80fed8625d3","Type":"ContainerDied","Data":"720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852"} Oct 31 16:05:14.939047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.938991 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c" event={"ID":"ebf94031-4801-4661-a79a-a80fed8625d3","Type":"ContainerDied","Data":"c21167f6a1fb41db9cca1d52abbd3f1d291e415fefdfdd4a75050f800e5a1f7b"} Oct 31 16:05:14.939047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.939019 4865 scope.go:122] "RemoveContainer" containerID="720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852" Oct 31 16:05:14.940377 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.940355 4865 generic.go:356] "Generic (PLEG): container finished" podID="5204b4e7-05da-4091-8727-83cb64939951" containerID="df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079" exitCode=0 Oct 31 16:05:14.940529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.940415 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" Oct 31 16:05:14.940529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.940436 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" event={"ID":"5204b4e7-05da-4091-8727-83cb64939951","Type":"ContainerDied","Data":"df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079"} Oct 31 16:05:14.940529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.940503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9" event={"ID":"5204b4e7-05da-4091-8727-83cb64939951","Type":"ContainerDied","Data":"67fb05685654084c03211238fd744edb2eebf0b46da599c502578c79a2b3bd23"} Oct 31 16:05:14.942176 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.942154 4865 generic.go:356] "Generic (PLEG): container finished" podID="c02b3b7a-88d8-4325-a130-0965a02c1109" containerID="b2c675b6dde40b6727a4e7672232dd93b92b84ece717761412a6285c26906064" exitCode=0 Oct 31 16:05:14.942297 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.942190 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" event={"ID":"c02b3b7a-88d8-4325-a130-0965a02c1109","Type":"ContainerDied","Data":"b2c675b6dde40b6727a4e7672232dd93b92b84ece717761412a6285c26906064"} Oct 31 16:05:14.949333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.949317 4865 scope.go:122] "RemoveContainer" containerID="720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852" Oct 31 16:05:14.949675 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:14.949639 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852\": container with ID starting with 720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852 not found: ID does not exist" containerID="720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852" Oct 31 16:05:14.949782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.949670 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852"} err="failed to get container status \"720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852\": rpc error: code = NotFound desc = could not find container \"720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852\": container with ID starting with 720cf16676bb9c0687511408016aaade2e3eb73246e635d1d362301506705852 not found: ID does not exist" Oct 31 16:05:14.949782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.949694 4865 scope.go:122] "RemoveContainer" containerID="df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079" Oct 31 16:05:14.958316 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.958298 4865 scope.go:122] "RemoveContainer" containerID="df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079" Oct 31 16:05:14.958585 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:14.958563 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079\": container with ID starting with df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079 not found: ID does not exist" containerID="df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079" Oct 31 16:05:14.958683 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:14.958592 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079"} err="failed to get container status \"df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079\": rpc error: code = NotFound desc = could not find container \"df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079\": container with ID starting with df1bf059a96f79bedfab1bfd46c1c8e6c567990dad067cebf5484d828c6eb079 not found: ID does not exist" Oct 31 16:05:15.105048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:15.104966 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9"] Oct 31 16:05:15.111606 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:15.111578 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e72453-6b467fd96c-2txt9"] Oct 31 16:05:15.223696 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:15.223661 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c"] Oct 31 16:05:15.238778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:15.238740 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e72453-55577fc69f-9tw5c"] Oct 31 16:05:15.993695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:15.993657 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" containerID="cri-o://92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e" gracePeriod=45 Oct 31 16:05:16.126490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.126448 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:16.275463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.275436 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 16:05:16.297685 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.297656 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-openshift-ca\") pod \"c02b3b7a-88d8-4325-a130-0965a02c1109\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " Oct 31 16:05:16.297828 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.297737 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-xks-ca\") pod \"c02b3b7a-88d8-4325-a130-0965a02c1109\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " Oct 31 16:05:16.297828 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.297784 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02b3b7a-88d8-4325-a130-0965a02c1109-kube-api-access-d6tsq\" (UniqueName: \"kubernetes.io/projected/c02b3b7a-88d8-4325-a130-0965a02c1109-kube-api-access-d6tsq\") pod \"c02b3b7a-88d8-4325-a130-0965a02c1109\" (UID: \"c02b3b7a-88d8-4325-a130-0965a02c1109\") " Oct 31 16:05:16.298097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.298067 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-openshift-ca" pod "c02b3b7a-88d8-4325-a130-0965a02c1109" (UID: "c02b3b7a-88d8-4325-a130-0965a02c1109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:16.298325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.298299 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.300050 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.300020 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02b3b7a-88d8-4325-a130-0965a02c1109-kube-api-access-d6tsq" pod "c02b3b7a-88d8-4325-a130-0965a02c1109" (UID: "c02b3b7a-88d8-4325-a130-0965a02c1109"). InnerVolumeSpecName "kube-api-access-d6tsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:16.300401 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.300379 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-xks-ca" pod "c02b3b7a-88d8-4325-a130-0965a02c1109" (UID: "c02b3b7a-88d8-4325-a130-0965a02c1109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:16.399708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.399672 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-home\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-home\") pod \"105da59e-13ba-48f4-a007-5be0f8203eeb\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " Oct 31 16:05:16.399895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.399769 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105da59e-13ba-48f4-a007-5be0f8203eeb-kube-api-access-sbk4v\" (UniqueName: \"kubernetes.io/projected/105da59e-13ba-48f4-a007-5be0f8203eeb-kube-api-access-sbk4v\") pod \"105da59e-13ba-48f4-a007-5be0f8203eeb\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " Oct 31 16:05:16.399895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.399815 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105da59e-13ba-48f4-a007-5be0f8203eeb-tls-certs\" (UniqueName: \"kubernetes.io/secret/105da59e-13ba-48f4-a007-5be0f8203eeb-tls-certs\") pod \"105da59e-13ba-48f4-a007-5be0f8203eeb\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " Oct 31 16:05:16.399895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.399880 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-tmp-dir\") pod \"105da59e-13ba-48f4-a007-5be0f8203eeb\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " Oct 31 16:05:16.400096 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.399936 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-model-cache\") pod \"105da59e-13ba-48f4-a007-5be0f8203eeb\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " Oct 31 16:05:16.400096 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.400034 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-dshm\") pod \"105da59e-13ba-48f4-a007-5be0f8203eeb\" (UID: \"105da59e-13ba-48f4-a007-5be0f8203eeb\") " Oct 31 16:05:16.400351 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.400321 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-home" pod "105da59e-13ba-48f4-a007-5be0f8203eeb" (UID: "105da59e-13ba-48f4-a007-5be0f8203eeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.400497 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.400455 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.400795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.400502 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c02b3b7a-88d8-4325-a130-0965a02c1109-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.400795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.400521 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tsq\" (UniqueName: \"kubernetes.io/projected/c02b3b7a-88d8-4325-a130-0965a02c1109-kube-api-access-d6tsq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.401087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.401059 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-tmp-dir" pod "105da59e-13ba-48f4-a007-5be0f8203eeb" (UID: "105da59e-13ba-48f4-a007-5be0f8203eeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.401087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.401076 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-model-cache" pod "105da59e-13ba-48f4-a007-5be0f8203eeb" (UID: "105da59e-13ba-48f4-a007-5be0f8203eeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.402422 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.402399 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105da59e-13ba-48f4-a007-5be0f8203eeb-tls-certs" pod "105da59e-13ba-48f4-a007-5be0f8203eeb" (UID: "105da59e-13ba-48f4-a007-5be0f8203eeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:16.402899 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.402874 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-dshm" pod "105da59e-13ba-48f4-a007-5be0f8203eeb" (UID: "105da59e-13ba-48f4-a007-5be0f8203eeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.403258 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.403236 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105da59e-13ba-48f4-a007-5be0f8203eeb-kube-api-access-sbk4v" pod "105da59e-13ba-48f4-a007-5be0f8203eeb" (UID: "105da59e-13ba-48f4-a007-5be0f8203eeb"). InnerVolumeSpecName "kube-api-access-sbk4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:16.496872 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.496837 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5204b4e7-05da-4091-8727-83cb64939951" path="/var/lib/kubelet/pods/5204b4e7-05da-4091-8727-83cb64939951/volumes" Oct 31 16:05:16.497330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.497309 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf94031-4801-4661-a79a-a80fed8625d3" path="/var/lib/kubelet/pods/ebf94031-4801-4661-a79a-a80fed8625d3/volumes" Oct 31 16:05:16.501341 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.501308 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.501453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.501344 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.501453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.501359 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105da59e-13ba-48f4-a007-5be0f8203eeb-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.501453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.501375 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbk4v\" (UniqueName: \"kubernetes.io/projected/105da59e-13ba-48f4-a007-5be0f8203eeb-kube-api-access-sbk4v\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.501453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.501389 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105da59e-13ba-48f4-a007-5be0f8203eeb-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.959708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.959676 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" Oct 31 16:05:16.959901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.959707 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf" event={"ID":"c02b3b7a-88d8-4325-a130-0965a02c1109","Type":"ContainerDied","Data":"94d5bcec8a0887486ae4ac07e1ac95d75b208fb4724b0bb4326abaa7a948ef17"} Oct 31 16:05:16.959901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.959761 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94d5bcec8a0887486ae4ac07e1ac95d75b208fb4724b0bb4326abaa7a948ef17" Oct 31 16:05:16.961145 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.961120 4865 generic.go:356] "Generic (PLEG): container finished" podID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerID="92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e" exitCode=0 Oct 31 16:05:16.961286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.961218 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" Oct 31 16:05:16.961286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.961255 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" event={"ID":"105da59e-13ba-48f4-a007-5be0f8203eeb","Type":"ContainerDied","Data":"92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e"} Oct 31 16:05:16.961389 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.961294 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk" event={"ID":"105da59e-13ba-48f4-a007-5be0f8203eeb","Type":"ContainerDied","Data":"9b4412c50c1c987d4feb327035672b62fcfe5280b14f8619aa25940e789365cf"} Oct 31 16:05:16.961389 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.961312 4865 scope.go:122] "RemoveContainer" containerID="92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e" Oct 31 16:05:16.970832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.970813 4865 scope.go:122] "RemoveContainer" containerID="92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e" Oct 31 16:05:16.971100 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:16.971081 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e\": container with ID starting with 92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e not found: ID does not exist" containerID="92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e" Oct 31 16:05:16.971148 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:16.971112 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e"} err="failed to get container status \"92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e\": rpc error: code = NotFound desc = could not find container \"92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e\": container with ID starting with 92485c175a1cb8dc1a6ef566ac286d04710f5dca6323ec5b9393df1a89318f6e not found: ID does not exist" Oct 31 16:05:17.182842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:17.182810 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk"] Oct 31 16:05:17.185643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:17.185624 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-190693/premium-w3-190693-kserve-6c766d86dc-49tzk"] Oct 31 16:05:17.414967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:17.414877 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" podUID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" containerName="payload-processing" containerID="cri-o://3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7" gracePeriod=25 Oct 31 16:05:17.489000 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:17.488961 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:05:17.498202 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:17.498177 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" podUID="65424c96-2c96-40c4-b781-ac6ace97bf35" containerName="payload-pre-processing" containerID="cri-o://a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f" gracePeriod=25 Oct 31 16:05:18.462613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:18.462502 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:18.495519 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:18.495491 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" path="/var/lib/kubelet/pods/105da59e-13ba-48f4-a007-5be0f8203eeb/volumes" Oct 31 16:05:18.625213 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:18.625180 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:18.748873 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:18.748789 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:19.480089 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:19.480055 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:22.154659 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.154626 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm"] Oct 31 16:05:22.155074 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155061 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ebf94031-4801-4661-a79a-a80fed8625d3" containerName="payload-pre-processing" Oct 31 16:05:22.155125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155075 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf94031-4801-4661-a79a-a80fed8625d3" containerName="payload-pre-processing" Oct 31 16:05:22.155125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155110 4865 cpu_manager.go:414] "containerMap: removing container" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" Oct 31 16:05:22.155125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155117 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" Oct 31 16:05:22.155234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155127 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5204b4e7-05da-4091-8727-83cb64939951" containerName="payload-processing" Oct 31 16:05:22.155234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155136 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5204b4e7-05da-4091-8727-83cb64939951" containerName="payload-processing" Oct 31 16:05:22.155234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155146 4865 cpu_manager.go:414] "containerMap: removing container" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" Oct 31 16:05:22.155234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155151 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" Oct 31 16:05:22.155234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155177 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c02b3b7a-88d8-4325-a130-0965a02c1109" containerName="revoke-keys" Oct 31 16:05:22.155234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155182 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02b3b7a-88d8-4325-a130-0965a02c1109" containerName="revoke-keys" Oct 31 16:05:22.155428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155251 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="105da59e-13ba-48f4-a007-5be0f8203eeb" containerName="main" Oct 31 16:05:22.155428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155260 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5204b4e7-05da-4091-8727-83cb64939951" containerName="payload-processing" Oct 31 16:05:22.155428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155270 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf89f0e4-3499-4802-9a05-72062f42def8" containerName="istio-proxy" Oct 31 16:05:22.155428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155276 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02b3b7a-88d8-4325-a130-0965a02c1109" containerName="revoke-keys" Oct 31 16:05:22.155428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.155283 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf94031-4801-4661-a79a-a80fed8625d3" containerName="payload-pre-processing" Oct 31 16:05:22.159820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.159804 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.213899 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.213868 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm"] Oct 31 16:05:22.262664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.262626 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:05:22.262664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.262647 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:05:22.262664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.262643 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:05:22.262912 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.262655 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:05:22.264147 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.264124 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.264210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.264189 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.264326 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.264308 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zq6z\" (UniqueName: \"kubernetes.io/projected/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-kube-api-access-6zq6z\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.365246 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.365207 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.365460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.365284 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.365460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.365319 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zq6z\" (UniqueName: \"kubernetes.io/projected/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-kube-api-access-6zq6z\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.365969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.365942 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.367842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.367823 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.423299 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.423228 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zq6z\" (UniqueName: \"kubernetes.io/projected/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-kube-api-access-6zq6z\") pod \"maas-api-revoke-sub-98ee7413-8twnm\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.473426 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.473398 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:22.846017 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:05:22.845985 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1ec719_741b_4bd3_9caf_8520bb32c7c9.slice/crio-3dce1b09a6ca148c187b0d9f0ea6de756e078f1c306dcc34fa7d63576d5fa27f WatchSource:0}: Error finding container 3dce1b09a6ca148c187b0d9f0ea6de756e078f1c306dcc34fa7d63576d5fa27f: Status 404 returned error can't find the container with id 3dce1b09a6ca148c187b0d9f0ea6de756e078f1c306dcc34fa7d63576d5fa27f Oct 31 16:05:22.846613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:22.846526 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm"] Oct 31 16:05:23.003131 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.003088 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" event={"ID":"bc1ec719-741b-4bd3-9caf-8520bb32c7c9","Type":"ContainerStarted","Data":"3dce1b09a6ca148c187b0d9f0ea6de756e078f1c306dcc34fa7d63576d5fa27f"} Oct 31 16:05:23.125627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.125594 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:23.332348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.332315 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6"] Oct 31 16:05:23.336173 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.336157 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.375857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.375790 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.375857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.375825 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.375857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.375843 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwq5j\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-kube-api-access-pwq5j\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.376054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.375960 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.376054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.376001 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.376168 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.376108 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.376168 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.376145 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.376281 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.376186 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.376281 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.376251 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.389910 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.389885 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-190693-openshift-default-dockercfg-qdhg8\"" Oct 31 16:05:23.444392 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.444359 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6"] Oct 31 16:05:23.477013 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.476986 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477028 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477060 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477075 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477114 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477134 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477148 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwq5j\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-kube-api-access-pwq5j\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477203 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477244 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477540 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477590 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477656 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477804 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.477983 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.477945 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.479389 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.479371 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.480226 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.480208 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.534964 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.534933 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.535106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.535062 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwq5j\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-kube-api-access-pwq5j\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-cbgg6\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.649647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.649621 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:23.750181 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:23.750143 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" podUID="65424c96-2c96-40c4-b781-ac6ace97bf35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:24.010314 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.010279 4865 generic.go:356] "Generic (PLEG): container finished" podID="bc1ec719-741b-4bd3-9caf-8520bb32c7c9" containerID="c56c0a78088b4b1964e2588de0b154c3b2eab412797014ecace1427fa4530d0c" exitCode=0 Oct 31 16:05:24.010524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.010316 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" event={"ID":"bc1ec719-741b-4bd3-9caf-8520bb32c7c9","Type":"ContainerDied","Data":"c56c0a78088b4b1964e2588de0b154c3b2eab412797014ecace1427fa4530d0c"} Oct 31 16:05:24.074734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.074709 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6"] Oct 31 16:05:24.075595 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:05:24.075567 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a39eca_feaa_4c2d_a545_3b279ba7b4bd.slice/crio-3977935ca04b300c7acac8576877626d9595a9ea9d7097fcdba01576a2e79275 WatchSource:0}: Error finding container 3977935ca04b300c7acac8576877626d9595a9ea9d7097fcdba01576a2e79275: Status 404 returned error can't find the container with id 3977935ca04b300c7acac8576877626d9595a9ea9d7097fcdba01576a2e79275 Oct 31 16:05:24.078069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.078037 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:05:24.078172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.078101 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:05:24.078172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.078130 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:05:24.754212 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.754169 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" podUID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:24.968923 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:24.968885 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:25.024552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.023661 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" event={"ID":"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd","Type":"ContainerStarted","Data":"a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e"} Oct 31 16:05:25.024552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.023725 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" event={"ID":"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd","Type":"ContainerStarted","Data":"3977935ca04b300c7acac8576877626d9595a9ea9d7097fcdba01576a2e79275"} Oct 31 16:05:25.186544 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.186520 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:25.192000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.191955 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podStartSLOduration=2.191940901 podStartE2EDuration="2.191940901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:05:25.190102499 +0000 UTC m=+4770.809653918" watchObservedRunningTime="2026-10-31 16:05:25.191940901 +0000 UTC m=+4770.811492312" Oct 31 16:05:25.218967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.218926 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-xks-ca\") pod \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " Oct 31 16:05:25.219175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.218990 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-openshift-ca\") pod \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " Oct 31 16:05:25.219175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.219044 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-kube-api-access-6zq6z\" (UniqueName: \"kubernetes.io/projected/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-kube-api-access-6zq6z\") pod \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\" (UID: \"bc1ec719-741b-4bd3-9caf-8520bb32c7c9\") " Oct 31 16:05:25.219753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.219727 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-openshift-ca" pod "bc1ec719-741b-4bd3-9caf-8520bb32c7c9" (UID: "bc1ec719-741b-4bd3-9caf-8520bb32c7c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:25.221203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.221172 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-xks-ca" pod "bc1ec719-741b-4bd3-9caf-8520bb32c7c9" (UID: "bc1ec719-741b-4bd3-9caf-8520bb32c7c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:25.221313 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.221295 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-kube-api-access-6zq6z" pod "bc1ec719-741b-4bd3-9caf-8520bb32c7c9" (UID: "bc1ec719-741b-4bd3-9caf-8520bb32c7c9"). InnerVolumeSpecName "kube-api-access-6zq6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:25.323797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.323281 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:25.323797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.323312 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:25.323797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.323322 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zq6z\" (UniqueName: \"kubernetes.io/projected/bc1ec719-741b-4bd3-9caf-8520bb32c7c9-kube-api-access-6zq6z\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:25.650227 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.650134 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:25.651571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.651550 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 16:05:25.651633 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:25.651595 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 16:05:26.029603 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:26.029571 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" Oct 31 16:05:26.030006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:26.029604 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm" event={"ID":"bc1ec719-741b-4bd3-9caf-8520bb32c7c9","Type":"ContainerDied","Data":"3dce1b09a6ca148c187b0d9f0ea6de756e078f1c306dcc34fa7d63576d5fa27f"} Oct 31 16:05:26.030006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:26.029653 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dce1b09a6ca148c187b0d9f0ea6de756e078f1c306dcc34fa7d63576d5fa27f" Oct 31 16:05:26.650054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:26.650018 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 16:05:26.650325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:26.650072 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 16:05:27.229216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:27.229175 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:27.650167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:27.650081 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 16:05:27.650506 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:27.650433 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 16:05:28.650570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:28.650528 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 16:05:28.651001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:28.650591 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 16:05:29.045825 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:29.045792 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:29.650327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:29.650294 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 16:05:29.650551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:29.650344 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 16:05:30.649878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:30.649841 4865 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" start-of-body= Oct 31 16:05:30.650271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:30.649898 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.125:15021/healthz/ready\": dial tcp 10.132.1.125:15021: connect: connection refused" Oct 31 16:05:31.784819 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:31.784784 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:31.785220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:31.785195 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:31.786019 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:31.785998 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:32.879072 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.879037 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd"] Oct 31 16:05:32.879536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.879522 4865 cpu_manager.go:414] "containerMap: removing container" podUID="bc1ec719-741b-4bd3-9caf-8520bb32c7c9" containerName="revoke-keys" Oct 31 16:05:32.879536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.879536 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1ec719-741b-4bd3-9caf-8520bb32c7c9" containerName="revoke-keys" Oct 31 16:05:32.879640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.879619 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1ec719-741b-4bd3-9caf-8520bb32c7c9" containerName="revoke-keys" Oct 31 16:05:32.883146 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.883130 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:32.939834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.939810 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd"] Oct 31 16:05:32.987533 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.987500 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:05:32.990425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.990401 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:05:32.990646 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.990623 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:05:32.990706 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:32.990690 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:05:33.007690 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.007663 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.007806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.007723 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.007847 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.007827 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2z9z\" (UniqueName: \"kubernetes.io/projected/3ba618bb-ec1f-4748-a68c-74002b195357-kube-api-access-l2z9z\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.108640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.108608 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.108794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.108655 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2z9z\" (UniqueName: \"kubernetes.io/projected/3ba618bb-ec1f-4748-a68c-74002b195357-kube-api-access-l2z9z\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.108794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.108767 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.109316 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.109288 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.111401 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.111375 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.166993 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.166923 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2z9z\" (UniqueName: \"kubernetes.io/projected/3ba618bb-ec1f-4748-a68c-74002b195357-kube-api-access-l2z9z\") pod \"maas-api-revoke-sub-6e1a9055-n8twd\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.195935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.195902 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:33.569355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.569319 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd"] Oct 31 16:05:33.569614 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:05:33.569589 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba618bb_ec1f_4748_a68c_74002b195357.slice/crio-28761b66d5c14ad9a385629c8ac43f7642332b6e3ffda1a847e9ddba9ed7ce15 WatchSource:0}: Error finding container 28761b66d5c14ad9a385629c8ac43f7642332b6e3ffda1a847e9ddba9ed7ce15: Status 404 returned error can't find the container with id 28761b66d5c14ad9a385629c8ac43f7642332b6e3ffda1a847e9ddba9ed7ce15 Oct 31 16:05:33.726413 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.726378 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:33.751202 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:33.751166 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" podUID="65424c96-2c96-40c4-b781-ac6ace97bf35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:34.078957 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:34.078915 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" event={"ID":"3ba618bb-ec1f-4748-a68c-74002b195357","Type":"ContainerStarted","Data":"a5f98a7e7beeaad2902736b3008166b8d508452087b6197c1300d41c392c67f1"} Oct 31 16:05:34.079421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:34.078972 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" event={"ID":"3ba618bb-ec1f-4748-a68c-74002b195357","Type":"ContainerStarted","Data":"28761b66d5c14ad9a385629c8ac43f7642332b6e3ffda1a847e9ddba9ed7ce15"} Oct 31 16:05:34.196297 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:34.196190 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" podStartSLOduration=2.196176367 podStartE2EDuration="2.196176367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:05:34.195320747 +0000 UTC m=+4779.814872159" watchObservedRunningTime="2026-10-31 16:05:34.196176367 +0000 UTC m=+4779.815727777" Oct 31 16:05:34.754120 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:34.754030 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" podUID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:35.086725 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:35.086640 4865 generic.go:356] "Generic (PLEG): container finished" podID="3ba618bb-ec1f-4748-a68c-74002b195357" containerID="a5f98a7e7beeaad2902736b3008166b8d508452087b6197c1300d41c392c67f1" exitCode=0 Oct 31 16:05:35.087111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:35.086723 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" event={"ID":"3ba618bb-ec1f-4748-a68c-74002b195357","Type":"ContainerDied","Data":"a5f98a7e7beeaad2902736b3008166b8d508452087b6197c1300d41c392c67f1"} Oct 31 16:05:36.272364 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.272336 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:36.348245 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.348167 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-xks-ca\") pod \"3ba618bb-ec1f-4748-a68c-74002b195357\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " Oct 31 16:05:36.348245 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.348224 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba618bb-ec1f-4748-a68c-74002b195357-kube-api-access-l2z9z\" (UniqueName: \"kubernetes.io/projected/3ba618bb-ec1f-4748-a68c-74002b195357-kube-api-access-l2z9z\") pod \"3ba618bb-ec1f-4748-a68c-74002b195357\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " Oct 31 16:05:36.348517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.348297 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-openshift-ca\") pod \"3ba618bb-ec1f-4748-a68c-74002b195357\" (UID: \"3ba618bb-ec1f-4748-a68c-74002b195357\") " Oct 31 16:05:36.348836 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.348803 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-openshift-ca" pod "3ba618bb-ec1f-4748-a68c-74002b195357" (UID: "3ba618bb-ec1f-4748-a68c-74002b195357"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:36.350660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.350615 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-xks-ca" pod "3ba618bb-ec1f-4748-a68c-74002b195357" (UID: "3ba618bb-ec1f-4748-a68c-74002b195357"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:36.351111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.351076 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba618bb-ec1f-4748-a68c-74002b195357-kube-api-access-l2z9z" pod "3ba618bb-ec1f-4748-a68c-74002b195357" (UID: "3ba618bb-ec1f-4748-a68c-74002b195357"). InnerVolumeSpecName "kube-api-access-l2z9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:36.450082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.450028 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:36.450082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.450087 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2z9z\" (UniqueName: \"kubernetes.io/projected/3ba618bb-ec1f-4748-a68c-74002b195357-kube-api-access-l2z9z\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:36.450322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:36.450107 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba618bb-ec1f-4748-a68c-74002b195357-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:37.100775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.100737 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" event={"ID":"3ba618bb-ec1f-4748-a68c-74002b195357","Type":"ContainerDied","Data":"28761b66d5c14ad9a385629c8ac43f7642332b6e3ffda1a847e9ddba9ed7ce15"} Oct 31 16:05:37.100775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.100779 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd" Oct 31 16:05:37.101062 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.100796 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28761b66d5c14ad9a385629c8ac43f7642332b6e3ffda1a847e9ddba9ed7ce15" Oct 31 16:05:37.188808 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.188773 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:37.323414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.323380 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:37.770598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.770571 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 16:05:37.792297 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.792274 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 16:05:37.870880 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.870850 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65424c96-2c96-40c4-b781-ac6ace97bf35-kube-api-access-j4bzn\" (UniqueName: \"kubernetes.io/projected/65424c96-2c96-40c4-b781-ac6ace97bf35-kube-api-access-j4bzn\") pod \"65424c96-2c96-40c4-b781-ac6ace97bf35\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " Oct 31 16:05:37.871091 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.870973 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-plugins-config-volume\") pod \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " Oct 31 16:05:37.871176 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.871105 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65424c96-2c96-40c4-b781-ac6ace97bf35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65424c96-2c96-40c4-b781-ac6ace97bf35-plugins-config-volume\") pod \"65424c96-2c96-40c4-b781-ac6ace97bf35\" (UID: \"65424c96-2c96-40c4-b781-ac6ace97bf35\") " Oct 31 16:05:37.871176 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.871147 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-kube-api-access-v75q5\" (UniqueName: \"kubernetes.io/projected/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-kube-api-access-v75q5\") pod \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\" (UID: \"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41\") " Oct 31 16:05:37.871476 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.871448 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65424c96-2c96-40c4-b781-ac6ace97bf35-plugins-config-volume" pod "65424c96-2c96-40c4-b781-ac6ace97bf35" (UID: "65424c96-2c96-40c4-b781-ac6ace97bf35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:37.871641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.871446 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-plugins-config-volume" pod "6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" (UID: "6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:37.871641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.871598 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65424c96-2c96-40c4-b781-ac6ace97bf35-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:37.873246 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.873226 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65424c96-2c96-40c4-b781-ac6ace97bf35-kube-api-access-j4bzn" pod "65424c96-2c96-40c4-b781-ac6ace97bf35" (UID: "65424c96-2c96-40c4-b781-ac6ace97bf35"). InnerVolumeSpecName "kube-api-access-j4bzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:37.873330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.873288 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-kube-api-access-v75q5" pod "6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" (UID: "6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41"). InnerVolumeSpecName "kube-api-access-v75q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:37.972278 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.972200 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4bzn\" (UniqueName: \"kubernetes.io/projected/65424c96-2c96-40c4-b781-ac6ace97bf35-kube-api-access-j4bzn\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:37.972278 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.972228 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:37.972278 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:37.972239 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75q5\" (UniqueName: \"kubernetes.io/projected/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41-kube-api-access-v75q5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:38.052749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.052718 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd"] Oct 31 16:05:38.055432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.055401 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-badaa017-wnjgd"] Oct 31 16:05:38.108808 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.108771 4865 generic.go:356] "Generic (PLEG): container finished" podID="65424c96-2c96-40c4-b781-ac6ace97bf35" containerID="a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f" exitCode=0 Oct 31 16:05:38.109005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.108812 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" Oct 31 16:05:38.109005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.108812 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" event={"ID":"65424c96-2c96-40c4-b781-ac6ace97bf35","Type":"ContainerDied","Data":"a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f"} Oct 31 16:05:38.109005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.108876 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2" event={"ID":"65424c96-2c96-40c4-b781-ac6ace97bf35","Type":"ContainerDied","Data":"7c4e5749da22cedc07046c6f329a270c928962445b73369f1df12a1da94436d6"} Oct 31 16:05:38.109005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.108909 4865 scope.go:122] "RemoveContainer" containerID="a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f" Oct 31 16:05:38.110755 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.110729 4865 generic.go:356] "Generic (PLEG): container finished" podID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" containerID="3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7" exitCode=0 Oct 31 16:05:38.110877 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.110777 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" Oct 31 16:05:38.110877 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.110807 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" event={"ID":"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41","Type":"ContainerDied","Data":"3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7"} Oct 31 16:05:38.110877 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.110859 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz" event={"ID":"6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41","Type":"ContainerDied","Data":"68169d60159be4707bcd9101080b8a57eb6fe068d19adf2da750bdea8fac295f"} Oct 31 16:05:38.120251 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.120219 4865 scope.go:122] "RemoveContainer" containerID="a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f" Oct 31 16:05:38.120641 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:38.120615 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f\": container with ID starting with a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f not found: ID does not exist" containerID="a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f" Oct 31 16:05:38.120739 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.120650 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f"} err="failed to get container status \"a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f\": rpc error: code = NotFound desc = could not find container \"a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f\": container with ID starting with a6eec51672db4aff3d04f5ea87274812bd115bccf2ddc2b09c2859e7f93e9f7f not found: ID does not exist" Oct 31 16:05:38.120739 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.120676 4865 scope.go:122] "RemoveContainer" containerID="3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7" Oct 31 16:05:38.129186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.129166 4865 scope.go:122] "RemoveContainer" containerID="3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7" Oct 31 16:05:38.129451 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:38.129433 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7\": container with ID starting with 3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7 not found: ID does not exist" containerID="3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7" Oct 31 16:05:38.129545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.129459 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7"} err="failed to get container status \"3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7\": rpc error: code = NotFound desc = could not find container \"3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7\": container with ID starting with 3224936b8670dab7ebfe436f4ec4220979fc0c15ccb9e7ee7da5c80bba60a6f7 not found: ID does not exist" Oct 31 16:05:38.280494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.280407 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2"] Oct 31 16:05:38.282863 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.282843 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-190693-76db6f84b7-dc2k2"] Oct 31 16:05:38.394000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.393949 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz"] Oct 31 16:05:38.397940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.397916 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-190693-66f9c6c557-vmlxz"] Oct 31 16:05:38.495602 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.495570 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c9a085-d4a7-4874-a6f7-ac9e127bbefb" path="/var/lib/kubelet/pods/55c9a085-d4a7-4874-a6f7-ac9e127bbefb/volumes" Oct 31 16:05:38.495963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.495950 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65424c96-2c96-40c4-b781-ac6ace97bf35" path="/var/lib/kubelet/pods/65424c96-2c96-40c4-b781-ac6ace97bf35/volumes" Oct 31 16:05:38.496285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.496274 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" path="/var/lib/kubelet/pods/6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41/volumes" Oct 31 16:05:38.519921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.519890 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:38.661916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.661832 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:38.894221 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:38.894190 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:39.029735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:39.029702 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:39.498065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:39.498000 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:39.628519 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:39.628433 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:40.253915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:40.253885 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:40.770852 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:40.770821 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:40.786464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:40.786436 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r"] Oct 31 16:05:40.786658 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:40.786639 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" containerID="cri-o://048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5" gracePeriod=30 Oct 31 16:05:41.568094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:41.568052 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:42.840729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:42.840695 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:44.645464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.645432 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll"] Oct 31 16:05:44.645905 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.645896 4865 cpu_manager.go:414] "containerMap: removing container" podUID="65424c96-2c96-40c4-b781-ac6ace97bf35" containerName="payload-pre-processing" Oct 31 16:05:44.645995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.645909 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65424c96-2c96-40c4-b781-ac6ace97bf35" containerName="payload-pre-processing" Oct 31 16:05:44.645995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.645926 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" containerName="payload-processing" Oct 31 16:05:44.645995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.645932 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" containerName="payload-processing" Oct 31 16:05:44.645995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.645963 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3ba618bb-ec1f-4748-a68c-74002b195357" containerName="revoke-keys" Oct 31 16:05:44.645995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.645970 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba618bb-ec1f-4748-a68c-74002b195357" containerName="revoke-keys" Oct 31 16:05:44.646185 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.646058 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="65424c96-2c96-40c4-b781-ac6ace97bf35" containerName="payload-pre-processing" Oct 31 16:05:44.646185 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.646067 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9b1a22-b5ce-4118-b9e8-64fb9b8e4b41" containerName="payload-processing" Oct 31 16:05:44.646185 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.646078 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba618bb-ec1f-4748-a68c-74002b195357" containerName="revoke-keys" Oct 31 16:05:44.650685 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.650665 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.671551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.671525 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:44.704536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.704514 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:05:44.704536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.704528 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:05:44.704694 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.704543 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:05:44.747812 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.747774 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbw6\" (UniqueName: \"kubernetes.io/projected/075e66dd-8540-472c-9636-d924b5c86b13-kube-api-access-ppbw6\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.748001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.747871 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.748001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.747908 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.755917 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.755884 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:05:44.758871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.758847 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll"] Oct 31 16:05:44.848680 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.848647 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbw6\" (UniqueName: \"kubernetes.io/projected/075e66dd-8540-472c-9636-d924b5c86b13-kube-api-access-ppbw6\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.848833 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.848689 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.848833 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.848710 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.849311 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.849287 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.851210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.851193 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.906963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.906894 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbw6\" (UniqueName: \"kubernetes.io/projected/075e66dd-8540-472c-9636-d924b5c86b13-kube-api-access-ppbw6\") pod \"maas-api-revoke-sub-5ff2e88e-wxqll\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:44.963191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:44.963163 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:45.120326 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:45.120278 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:45.251571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:45.251539 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:45.337260 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:45.337236 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll"] Oct 31 16:05:45.338394 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:05:45.338366 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075e66dd_8540_472c_9636_d924b5c86b13.slice/crio-627a682c1a2806301ebbef49bed8aa4b8144bddb97ba6e4c84bbc4c9f1f88b51 WatchSource:0}: Error finding container 627a682c1a2806301ebbef49bed8aa4b8144bddb97ba6e4c84bbc4c9f1f88b51: Status 404 returned error can't find the container with id 627a682c1a2806301ebbef49bed8aa4b8144bddb97ba6e4c84bbc4c9f1f88b51 Oct 31 16:05:46.053304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.053250 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 16:05:46.162628 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162604 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318ae492-f923-4efd-b588-108ea07bee2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318ae492-f923-4efd-b588-108ea07bee2f-istiod-ca-cert\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.162754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162663 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-istio-token\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-istio-token\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.162754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162716 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-envoy\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.162840 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162791 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-kube-api-access-z55rj\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-kube-api-access-z55rj\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.162840 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162829 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-credential-socket\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.162950 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162889 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-certs\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.162950 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162922 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/318ae492-f923-4efd-b588-108ea07bee2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318ae492-f923-4efd-b588-108ea07bee2f-istio-podinfo\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.163065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162958 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-socket\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.163065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.162992 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-data\") pod \"318ae492-f923-4efd-b588-108ea07bee2f\" (UID: \"318ae492-f923-4efd-b588-108ea07bee2f\") " Oct 31 16:05:46.163065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.163032 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318ae492-f923-4efd-b588-108ea07bee2f-istiod-ca-cert" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:46.163239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.163152 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-credential-socket" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:46.163405 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.163382 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.163634 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.163410 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318ae492-f923-4efd-b588-108ea07bee2f-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.163634 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.163537 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-data" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:46.163634 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.163605 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-certs" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:46.163856 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.163750 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-socket" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:46.165202 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165174 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-istio-token" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:46.165303 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165211 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-envoy" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:46.165303 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165226 4865 generic.go:356] "Generic (PLEG): container finished" podID="318ae492-f923-4efd-b588-108ea07bee2f" containerID="048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5" exitCode=0 Oct 31 16:05:46.165303 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165280 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" Oct 31 16:05:46.165515 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165304 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" event={"ID":"318ae492-f923-4efd-b588-108ea07bee2f","Type":"ContainerDied","Data":"048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5"} Oct 31 16:05:46.165515 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165360 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r" event={"ID":"318ae492-f923-4efd-b588-108ea07bee2f","Type":"ContainerDied","Data":"06a9b519f18e0bcf81ab86806f53e9754b21af38f236f91488aa8716e1c1b8df"} Oct 31 16:05:46.165515 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165390 4865 scope.go:122] "RemoveContainer" containerID="048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5" Oct 31 16:05:46.165788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165766 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/318ae492-f923-4efd-b588-108ea07bee2f-istio-podinfo" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:05:46.165861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.165815 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-kube-api-access-z55rj" pod "318ae492-f923-4efd-b588-108ea07bee2f" (UID: "318ae492-f923-4efd-b588-108ea07bee2f"). InnerVolumeSpecName "kube-api-access-z55rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:46.167200 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.167181 4865 generic.go:356] "Generic (PLEG): container finished" podID="075e66dd-8540-472c-9636-d924b5c86b13" containerID="9c30c1bf67af6b72b3115276f5b864e095f33e445e1ce2ba9ff8a30f029a042d" exitCode=0 Oct 31 16:05:46.167292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.167260 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" event={"ID":"075e66dd-8540-472c-9636-d924b5c86b13","Type":"ContainerDied","Data":"9c30c1bf67af6b72b3115276f5b864e095f33e445e1ce2ba9ff8a30f029a042d"} Oct 31 16:05:46.167334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.167302 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" event={"ID":"075e66dd-8540-472c-9636-d924b5c86b13","Type":"ContainerStarted","Data":"627a682c1a2806301ebbef49bed8aa4b8144bddb97ba6e4c84bbc4c9f1f88b51"} Oct 31 16:05:46.231813 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.231792 4865 scope.go:122] "RemoveContainer" containerID="1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9" Oct 31 16:05:46.243297 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.243273 4865 scope.go:122] "RemoveContainer" containerID="048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5" Oct 31 16:05:46.243595 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:46.243573 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5\": container with ID starting with 048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5 not found: ID does not exist" containerID="048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5" Oct 31 16:05:46.243695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.243609 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5"} err="failed to get container status \"048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5\": rpc error: code = NotFound desc = could not find container \"048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5\": container with ID starting with 048d3ae5a2612e26afbe600e67a94592a2930c1fd93eacd4265dce0e5cf7d0e5 not found: ID does not exist" Oct 31 16:05:46.243695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.243634 4865 scope.go:122] "RemoveContainer" containerID="1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9" Oct 31 16:05:46.243961 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:46.243944 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9\": container with ID starting with 1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9 not found: ID does not exist" containerID="1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9" Oct 31 16:05:46.244013 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.243968 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9"} err="failed to get container status \"1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9\": rpc error: code = NotFound desc = could not find container \"1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9\": container with ID starting with 1e700528ba15e3b3f60a5a94f71246a010e3b5488a708642da7d47054c63b6a9 not found: ID does not exist" Oct 31 16:05:46.264632 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.264601 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.264632 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.264628 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318ae492-f923-4efd-b588-108ea07bee2f-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.264784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.264643 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.264784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.264656 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.264784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.264667 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.264784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.264680 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318ae492-f923-4efd-b588-108ea07bee2f-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.264784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.264692 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z55rj\" (UniqueName: \"kubernetes.io/projected/318ae492-f923-4efd-b588-108ea07bee2f-kube-api-access-z55rj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:46.591191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.591162 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r"] Oct 31 16:05:46.596887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:46.596859 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e72453-openshift-default-5bdf7bf485-4qn5r"] Oct 31 16:05:47.320777 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.320753 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:47.405239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.405208 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:47.475552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.475523 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-openshift-ca\") pod \"075e66dd-8540-472c-9636-d924b5c86b13\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " Oct 31 16:05:47.475732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.475606 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-xks-ca\") pod \"075e66dd-8540-472c-9636-d924b5c86b13\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " Oct 31 16:05:47.475732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.475682 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075e66dd-8540-472c-9636-d924b5c86b13-kube-api-access-ppbw6\" (UniqueName: \"kubernetes.io/projected/075e66dd-8540-472c-9636-d924b5c86b13-kube-api-access-ppbw6\") pod \"075e66dd-8540-472c-9636-d924b5c86b13\" (UID: \"075e66dd-8540-472c-9636-d924b5c86b13\") " Oct 31 16:05:47.476091 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.476061 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-openshift-ca" pod "075e66dd-8540-472c-9636-d924b5c86b13" (UID: "075e66dd-8540-472c-9636-d924b5c86b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:47.477780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.477752 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-xks-ca" pod "075e66dd-8540-472c-9636-d924b5c86b13" (UID: "075e66dd-8540-472c-9636-d924b5c86b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:47.478522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.478464 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075e66dd-8540-472c-9636-d924b5c86b13-kube-api-access-ppbw6" pod "075e66dd-8540-472c-9636-d924b5c86b13" (UID: "075e66dd-8540-472c-9636-d924b5c86b13"). InnerVolumeSpecName "kube-api-access-ppbw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:47.490848 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:47.490816 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:05:47.576604 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.576579 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:47.576604 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.576602 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075e66dd-8540-472c-9636-d924b5c86b13-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:47.576762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:47.576613 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbw6\" (UniqueName: \"kubernetes.io/projected/075e66dd-8540-472c-9636-d924b5c86b13-kube-api-access-ppbw6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:48.156366 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.156328 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt"] Oct 31 16:05:48.181948 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.181921 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" Oct 31 16:05:48.182146 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.181967 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll" event={"ID":"075e66dd-8540-472c-9636-d924b5c86b13","Type":"ContainerDied","Data":"627a682c1a2806301ebbef49bed8aa4b8144bddb97ba6e4c84bbc4c9f1f88b51"} Oct 31 16:05:48.182146 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.182012 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="627a682c1a2806301ebbef49bed8aa4b8144bddb97ba6e4c84bbc4c9f1f88b51" Oct 31 16:05:48.264584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.264552 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d"] Oct 31 16:05:48.496271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.496232 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318ae492-f923-4efd-b588-108ea07bee2f" path="/var/lib/kubelet/pods/318ae492-f923-4efd-b588-108ea07bee2f/volumes" Oct 31 16:05:48.592247 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.592214 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:48.715594 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:48.715547 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:49.404191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:49.404147 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:49.539944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:49.539906 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:49.680945 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:49.680796 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:50.872031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:50.871969 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:51.349561 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:51.349519 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:51.354952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:51.354919 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6"] Oct 31 16:05:51.355171 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:51.355151 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" containerID="cri-o://a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e" gracePeriod=30 Oct 31 16:05:52.885255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:52.885222 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:53.157557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:53.157445 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" podUID="6951eea9-0a8d-4278-88e0-982426951c8c" containerName="payload-processing" containerID="cri-o://6f055f551c0688ec0bbbb2e35b25278ee2b626059e35e092dc373b4a1746ec21" gracePeriod=25 Oct 31 16:05:53.264924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:53.264886 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" podUID="52f1e816-7f06-4c01-9d69-eb51b87787e3" containerName="payload-pre-processing" containerID="cri-o://96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2" gracePeriod=25 Oct 31 16:05:54.595084 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:54.595020 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:55.450226 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450193 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z"] Oct 31 16:05:55.450701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450686 4865 cpu_manager.go:414] "containerMap: removing container" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" Oct 31 16:05:55.450753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450702 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" Oct 31 16:05:55.450753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450713 4865 cpu_manager.go:414] "containerMap: removing container" podUID="075e66dd-8540-472c-9636-d924b5c86b13" containerName="revoke-keys" Oct 31 16:05:55.450753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450718 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075e66dd-8540-472c-9636-d924b5c86b13" containerName="revoke-keys" Oct 31 16:05:55.450753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450727 4865 cpu_manager.go:414] "containerMap: removing container" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" Oct 31 16:05:55.450753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450731 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" Oct 31 16:05:55.450922 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450816 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" Oct 31 16:05:55.450922 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450829 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="075e66dd-8540-472c-9636-d924b5c86b13" containerName="revoke-keys" Oct 31 16:05:55.450922 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.450838 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="318ae492-f923-4efd-b588-108ea07bee2f" containerName="istio-proxy" Oct 31 16:05:55.455294 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.455271 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.510735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.510703 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z"] Oct 31 16:05:55.557959 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.557930 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:05:55.557959 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.557945 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:05:55.557959 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.557959 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:05:55.558284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.557975 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:05:55.560512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.560461 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.560616 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.560599 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.560666 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.560643 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdkq\" (UniqueName: \"kubernetes.io/projected/4ec13819-1167-4dd6-9314-117272a8ad22-kube-api-access-qfdkq\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.661410 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.661381 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.661884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.661450 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.661884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.661512 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdkq\" (UniqueName: \"kubernetes.io/projected/4ec13819-1167-4dd6-9314-117272a8ad22-kube-api-access-qfdkq\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.662155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.662133 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.664040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.664020 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.719817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.719791 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdkq\" (UniqueName: \"kubernetes.io/projected/4ec13819-1167-4dd6-9314-117272a8ad22-kube-api-access-qfdkq\") pod \"maas-api-revoke-sub-214aae44-ppd9z\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:55.768155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:55.768127 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:56.044496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.044407 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x"] Oct 31 16:05:56.046781 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.046760 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb24cc7f-d2r9x"] Oct 31 16:05:56.154132 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.154102 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z"] Oct 31 16:05:56.235538 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.235505 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" event={"ID":"4ec13819-1167-4dd6-9314-117272a8ad22","Type":"ContainerStarted","Data":"3f7ba2dcbbcf16b40cf0033ff995645a29b73eba6342e96781d0756300f019e1"} Oct 31 16:05:56.255884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.255861 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:05:56.263501 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.263463 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:05:56.495502 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.495450 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b30efe-778d-453b-aaf4-010f66fdcdf9" path="/var/lib/kubelet/pods/74b30efe-778d-453b-aaf4-010f66fdcdf9/volumes" Oct 31 16:05:56.516357 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.516326 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" podUID="52f1e816-7f06-4c01-9d69-eb51b87787e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:56.516357 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.516341 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" podUID="6951eea9-0a8d-4278-88e0-982426951c8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:05:56.591169 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.591145 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:56.674818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.674784 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-envoy\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.674845 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istiod-ca-cert\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675008 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-socket\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675064 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-token\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-token\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675102 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-podinfo\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675133 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-data\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675176 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-credential-socket\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675234 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-kube-api-access-pwq5j\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-kube-api-access-pwq5j\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675261 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-certs\") pod \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\" (UID: \"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd\") " Oct 31 16:05:56.675703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675288 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istiod-ca-cert" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:56.675832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675708 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-credential-socket" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.675832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675717 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-data" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.675946 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675893 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-socket" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.675946 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675905 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.675946 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675925 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.676124 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.675993 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-certs" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.677597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.677565 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-token" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:56.677799 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.677779 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-kube-api-access-pwq5j" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "kube-api-access-pwq5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:56.678661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.678637 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-podinfo" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:05:56.678785 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.678753 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-envoy" pod "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" (UID: "f8a39eca-feaa-4c2d-a545-3b279ba7b4bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.776867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.776832 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.776867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.776860 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.776867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.776872 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.777122 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.776882 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.777122 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.776892 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwq5j\" (UniqueName: \"kubernetes.io/projected/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-kube-api-access-pwq5j\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.777122 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.776901 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.777122 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:56.776915 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:57.243505 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.243399 4865 generic.go:356] "Generic (PLEG): container finished" podID="4ec13819-1167-4dd6-9314-117272a8ad22" containerID="1b59652f18c07fbf803b64fdca74796d0008c5dcf6e4e237875f97ed86cbe754" exitCode=0 Oct 31 16:05:57.243655 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.243505 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" event={"ID":"4ec13819-1167-4dd6-9314-117272a8ad22","Type":"ContainerDied","Data":"1b59652f18c07fbf803b64fdca74796d0008c5dcf6e4e237875f97ed86cbe754"} Oct 31 16:05:57.244846 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.244822 4865 generic.go:356] "Generic (PLEG): container finished" podID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerID="a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e" exitCode=0 Oct 31 16:05:57.244967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.244860 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" Oct 31 16:05:57.244967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.244892 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" event={"ID":"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd","Type":"ContainerDied","Data":"a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e"} Oct 31 16:05:57.244967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.244933 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6" event={"ID":"f8a39eca-feaa-4c2d-a545-3b279ba7b4bd","Type":"ContainerDied","Data":"3977935ca04b300c7acac8576877626d9595a9ea9d7097fcdba01576a2e79275"} Oct 31 16:05:57.244967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.244954 4865 scope.go:122] "RemoveContainer" containerID="a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e" Oct 31 16:05:57.255029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.255009 4865 scope.go:122] "RemoveContainer" containerID="a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e" Oct 31 16:05:57.255317 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:05:57.255299 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e\": container with ID starting with a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e not found: ID does not exist" containerID="a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e" Oct 31 16:05:57.255397 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.255323 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e"} err="failed to get container status \"a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e\": rpc error: code = NotFound desc = could not find container \"a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e\": container with ID starting with a6e5b261693142051bf0e2243a5ec0ba547b04f3e9be153abb74a3934705855e not found: ID does not exist" Oct 31 16:05:57.473165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.473127 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6"] Oct 31 16:05:57.524287 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:57.524216 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-cbgg6"] Oct 31 16:05:58.397939 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.397911 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:58.493759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.493728 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec13819-1167-4dd6-9314-117272a8ad22-kube-api-access-qfdkq\" (UniqueName: \"kubernetes.io/projected/4ec13819-1167-4dd6-9314-117272a8ad22-kube-api-access-qfdkq\") pod \"4ec13819-1167-4dd6-9314-117272a8ad22\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " Oct 31 16:05:58.493937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.493787 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-openshift-ca\") pod \"4ec13819-1167-4dd6-9314-117272a8ad22\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " Oct 31 16:05:58.493937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.493829 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-xks-ca\") pod \"4ec13819-1167-4dd6-9314-117272a8ad22\" (UID: \"4ec13819-1167-4dd6-9314-117272a8ad22\") " Oct 31 16:05:58.494132 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.494108 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-openshift-ca" pod "4ec13819-1167-4dd6-9314-117272a8ad22" (UID: "4ec13819-1167-4dd6-9314-117272a8ad22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:58.494235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.494214 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:58.496099 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.496069 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-xks-ca" pod "4ec13819-1167-4dd6-9314-117272a8ad22" (UID: "4ec13819-1167-4dd6-9314-117272a8ad22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:58.496191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.496097 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec13819-1167-4dd6-9314-117272a8ad22-kube-api-access-qfdkq" pod "4ec13819-1167-4dd6-9314-117272a8ad22" (UID: "4ec13819-1167-4dd6-9314-117272a8ad22"). InnerVolumeSpecName "kube-api-access-qfdkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:58.496859 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.496839 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" path="/var/lib/kubelet/pods/f8a39eca-feaa-4c2d-a545-3b279ba7b4bd/volumes" Oct 31 16:05:58.595032 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.594996 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdkq\" (UniqueName: \"kubernetes.io/projected/4ec13819-1167-4dd6-9314-117272a8ad22-kube-api-access-qfdkq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:58.595032 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.595032 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec13819-1167-4dd6-9314-117272a8ad22-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:58.658758 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.658727 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:58.713288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.713262 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:05:58.721435 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.721414 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:05:58.954071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:58.953986 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:05:59.260263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:59.260231 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" Oct 31 16:05:59.260507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:59.260271 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z" event={"ID":"4ec13819-1167-4dd6-9314-117272a8ad22","Type":"ContainerDied","Data":"3f7ba2dcbbcf16b40cf0033ff995645a29b73eba6342e96781d0756300f019e1"} Oct 31 16:05:59.260507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:05:59.260319 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f7ba2dcbbcf16b40cf0033ff995645a29b73eba6342e96781d0756300f019e1" Oct 31 16:06:00.438364 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:00.438281 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:05.091753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:05.091719 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:06.515916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:06.515879 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" podUID="6951eea9-0a8d-4278-88e0-982426951c8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:06:06.516352 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:06.515879 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" podUID="52f1e816-7f06-4c01-9d69-eb51b87787e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:06:07.159031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:07.158977 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:07.291202 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:07.291168 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:08.702059 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:08.702017 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:09.190302 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:09.190242 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:09.190938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:09.190914 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv"] Oct 31 16:06:09.191170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:09.191139 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" containerID="cri-o://d1b6aade064ad7c4a53c1d563c4c156415727ada83484d664a2cf14df243a1d9" gracePeriod=30 Oct 31 16:06:09.658581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:09.658540 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:09.946144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:09.946064 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:10.295376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.295333 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5"] Oct 31 16:06:10.295790 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.295776 4865 cpu_manager.go:414] "containerMap: removing container" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" Oct 31 16:06:10.295838 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.295792 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" Oct 31 16:06:10.295838 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.295816 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4ec13819-1167-4dd6-9314-117272a8ad22" containerName="revoke-keys" Oct 31 16:06:10.295838 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.295821 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec13819-1167-4dd6-9314-117272a8ad22" containerName="revoke-keys" Oct 31 16:06:10.295962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.295899 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a39eca-feaa-4c2d-a545-3b279ba7b4bd" containerName="istio-proxy" Oct 31 16:06:10.295962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.295910 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec13819-1167-4dd6-9314-117272a8ad22" containerName="revoke-keys" Oct 31 16:06:10.299197 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.299180 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.353152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.353120 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:06:10.353393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.353153 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:06:10.353541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.353248 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:06:10.353541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.353375 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:06:10.354426 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.354405 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5"] Oct 31 16:06:10.426944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.426907 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.427152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.427006 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.427152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.427140 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zscp\" (UniqueName: \"kubernetes.io/projected/9f789656-052e-4c05-acef-cca7eaf9d108-kube-api-access-2zscp\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.528515 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.528457 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.528666 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.528568 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zscp\" (UniqueName: \"kubernetes.io/projected/9f789656-052e-4c05-acef-cca7eaf9d108-kube-api-access-2zscp\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.528666 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.528618 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.529274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.529204 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.531094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.531063 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.586974 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.586902 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zscp\" (UniqueName: \"kubernetes.io/projected/9f789656-052e-4c05-acef-cca7eaf9d108-kube-api-access-2zscp\") pod \"maas-api-revoke-sub-0d1c8dc8-q6ld5\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.612034 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.612008 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:10.985983 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:10.985950 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5"] Oct 31 16:06:10.986821 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:06:10.986785 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f789656_052e_4c05_acef_cca7eaf9d108.slice/crio-ec1cdd31ee987b088223a3bd137ee35649bdcfa131dfe38a156dd124eeaf0772 WatchSource:0}: Error finding container ec1cdd31ee987b088223a3bd137ee35649bdcfa131dfe38a156dd124eeaf0772: Status 404 returned error can't find the container with id ec1cdd31ee987b088223a3bd137ee35649bdcfa131dfe38a156dd124eeaf0772 Oct 31 16:06:11.328199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:11.328169 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" event={"ID":"9f789656-052e-4c05-acef-cca7eaf9d108","Type":"ContainerStarted","Data":"bb8fb11e92d6a493b9e37643315d919c8f751658c580b4309322e9fa4cc30dc4"} Oct 31 16:06:11.328311 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:11.328221 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" event={"ID":"9f789656-052e-4c05-acef-cca7eaf9d108","Type":"ContainerStarted","Data":"ec1cdd31ee987b088223a3bd137ee35649bdcfa131dfe38a156dd124eeaf0772"} Oct 31 16:06:11.443329 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:11.443275 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" podStartSLOduration=1.443262184 podStartE2EDuration="1.443262184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:11.44274231 +0000 UTC m=+4817.062293721" watchObservedRunningTime="2026-10-31 16:06:11.443262184 +0000 UTC m=+4817.062813643" Oct 31 16:06:12.335954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:12.335919 4865 generic.go:356] "Generic (PLEG): container finished" podID="9f789656-052e-4c05-acef-cca7eaf9d108" containerID="bb8fb11e92d6a493b9e37643315d919c8f751658c580b4309322e9fa4cc30dc4" exitCode=0 Oct 31 16:06:12.336380 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:12.335963 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" event={"ID":"9f789656-052e-4c05-acef-cca7eaf9d108","Type":"ContainerDied","Data":"bb8fb11e92d6a493b9e37643315d919c8f751658c580b4309322e9fa4cc30dc4"} Oct 31 16:06:12.462107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:12.462065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:13.344017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.343986 4865 generic.go:356] "Generic (PLEG): container finished" podID="6951eea9-0a8d-4278-88e0-982426951c8c" containerID="6f055f551c0688ec0bbbb2e35b25278ee2b626059e35e092dc373b4a1746ec21" exitCode=0 Oct 31 16:06:13.344512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.344054 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" event={"ID":"6951eea9-0a8d-4278-88e0-982426951c8c","Type":"ContainerDied","Data":"6f055f551c0688ec0bbbb2e35b25278ee2b626059e35e092dc373b4a1746ec21"} Oct 31 16:06:13.425077 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.424224 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:13.427526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.427499 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 16:06:13.484113 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.484090 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:13.528126 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.528102 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 16:06:13.557985 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.557951 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6951eea9-0a8d-4278-88e0-982426951c8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6951eea9-0a8d-4278-88e0-982426951c8c-plugins-config-volume\") pod \"6951eea9-0a8d-4278-88e0-982426951c8c\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " Oct 31 16:06:13.558133 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.558037 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f789656-052e-4c05-acef-cca7eaf9d108-kube-api-access-2zscp\" (UniqueName: \"kubernetes.io/projected/9f789656-052e-4c05-acef-cca7eaf9d108-kube-api-access-2zscp\") pod \"9f789656-052e-4c05-acef-cca7eaf9d108\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " Oct 31 16:06:13.558133 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.558109 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-openshift-ca\") pod \"9f789656-052e-4c05-acef-cca7eaf9d108\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " Oct 31 16:06:13.558286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.558169 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6951eea9-0a8d-4278-88e0-982426951c8c-kube-api-access-n4swd\" (UniqueName: \"kubernetes.io/projected/6951eea9-0a8d-4278-88e0-982426951c8c-kube-api-access-n4swd\") pod \"6951eea9-0a8d-4278-88e0-982426951c8c\" (UID: \"6951eea9-0a8d-4278-88e0-982426951c8c\") " Oct 31 16:06:13.558414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.558396 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-xks-ca\") pod \"9f789656-052e-4c05-acef-cca7eaf9d108\" (UID: \"9f789656-052e-4c05-acef-cca7eaf9d108\") " Oct 31 16:06:13.558576 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.558548 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-openshift-ca" pod "9f789656-052e-4c05-acef-cca7eaf9d108" (UID: "9f789656-052e-4c05-acef-cca7eaf9d108"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:06:13.558714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.558669 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6951eea9-0a8d-4278-88e0-982426951c8c-plugins-config-volume" pod "6951eea9-0a8d-4278-88e0-982426951c8c" (UID: "6951eea9-0a8d-4278-88e0-982426951c8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:06:13.559355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.559330 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6951eea9-0a8d-4278-88e0-982426951c8c-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:13.559355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.559362 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:13.560981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.560951 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6951eea9-0a8d-4278-88e0-982426951c8c-kube-api-access-n4swd" pod "6951eea9-0a8d-4278-88e0-982426951c8c" (UID: "6951eea9-0a8d-4278-88e0-982426951c8c"). InnerVolumeSpecName "kube-api-access-n4swd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:13.561087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.560986 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f789656-052e-4c05-acef-cca7eaf9d108-kube-api-access-2zscp" pod "9f789656-052e-4c05-acef-cca7eaf9d108" (UID: "9f789656-052e-4c05-acef-cca7eaf9d108"). InnerVolumeSpecName "kube-api-access-2zscp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:13.561087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.561020 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-xks-ca" pod "9f789656-052e-4c05-acef-cca7eaf9d108" (UID: "9f789656-052e-4c05-acef-cca7eaf9d108"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:06:13.660160 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.660122 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f1e816-7f06-4c01-9d69-eb51b87787e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f1e816-7f06-4c01-9d69-eb51b87787e3-plugins-config-volume\") pod \"52f1e816-7f06-4c01-9d69-eb51b87787e3\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " Oct 31 16:06:13.660339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.660312 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f1e816-7f06-4c01-9d69-eb51b87787e3-kube-api-access-gq5qq\" (UniqueName: \"kubernetes.io/projected/52f1e816-7f06-4c01-9d69-eb51b87787e3-kube-api-access-gq5qq\") pod \"52f1e816-7f06-4c01-9d69-eb51b87787e3\" (UID: \"52f1e816-7f06-4c01-9d69-eb51b87787e3\") " Oct 31 16:06:13.660692 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.660666 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f1e816-7f06-4c01-9d69-eb51b87787e3-plugins-config-volume" pod "52f1e816-7f06-4c01-9d69-eb51b87787e3" (UID: "52f1e816-7f06-4c01-9d69-eb51b87787e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:06:13.660843 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.660783 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4swd\" (UniqueName: \"kubernetes.io/projected/6951eea9-0a8d-4278-88e0-982426951c8c-kube-api-access-n4swd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:13.660914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.660851 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f789656-052e-4c05-acef-cca7eaf9d108-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:13.660914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.660870 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zscp\" (UniqueName: \"kubernetes.io/projected/9f789656-052e-4c05-acef-cca7eaf9d108-kube-api-access-2zscp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:13.663164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.663134 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f1e816-7f06-4c01-9d69-eb51b87787e3-kube-api-access-gq5qq" pod "52f1e816-7f06-4c01-9d69-eb51b87787e3" (UID: "52f1e816-7f06-4c01-9d69-eb51b87787e3"). InnerVolumeSpecName "kube-api-access-gq5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:13.761928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.761897 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f1e816-7f06-4c01-9d69-eb51b87787e3-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:13.761928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.761926 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5qq\" (UniqueName: \"kubernetes.io/projected/52f1e816-7f06-4c01-9d69-eb51b87787e3-kube-api-access-gq5qq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:13.960571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:13.960540 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:14.351798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.351766 4865 generic.go:356] "Generic (PLEG): container finished" podID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerID="d1b6aade064ad7c4a53c1d563c4c156415727ada83484d664a2cf14df243a1d9" exitCode=0 Oct 31 16:06:14.352267 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.351833 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" event={"ID":"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1","Type":"ContainerDied","Data":"d1b6aade064ad7c4a53c1d563c4c156415727ada83484d664a2cf14df243a1d9"} Oct 31 16:06:14.352267 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.351929 4865 scope.go:122] "RemoveContainer" containerID="bd51cd0fecb5c65f1f7c455ff55ebffc7c12d2e280c649e8be8ed8704c7bac1c" Oct 31 16:06:14.353420 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.353388 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" Oct 31 16:06:14.353547 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.353391 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5" event={"ID":"9f789656-052e-4c05-acef-cca7eaf9d108","Type":"ContainerDied","Data":"ec1cdd31ee987b088223a3bd137ee35649bdcfa131dfe38a156dd124eeaf0772"} Oct 31 16:06:14.353547 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.353524 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec1cdd31ee987b088223a3bd137ee35649bdcfa131dfe38a156dd124eeaf0772" Oct 31 16:06:14.355275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.355052 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" event={"ID":"6951eea9-0a8d-4278-88e0-982426951c8c","Type":"ContainerDied","Data":"28b0f03df3ecfadc5afef9e5c025fff23b6bebe85cab6ae18734cc744c63d8e7"} Oct 31 16:06:14.355275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.355081 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt" Oct 31 16:06:14.370754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.364191 4865 generic.go:356] "Generic (PLEG): container finished" podID="52f1e816-7f06-4c01-9d69-eb51b87787e3" containerID="96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2" exitCode=0 Oct 31 16:06:14.370754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.364247 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" event={"ID":"52f1e816-7f06-4c01-9d69-eb51b87787e3","Type":"ContainerDied","Data":"96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2"} Oct 31 16:06:14.370754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.364349 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" event={"ID":"52f1e816-7f06-4c01-9d69-eb51b87787e3","Type":"ContainerDied","Data":"f2edc69870b89c1d1911e6f6af6508e9f3018c9804df1711e726c3d0ab7f51b7"} Oct 31 16:06:14.370754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.364592 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d" Oct 31 16:06:14.370754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.364760 4865 scope.go:122] "RemoveContainer" containerID="6f055f551c0688ec0bbbb2e35b25278ee2b626059e35e092dc373b4a1746ec21" Oct 31 16:06:14.382825 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.382682 4865 scope.go:122] "RemoveContainer" containerID="96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2" Oct 31 16:06:14.392193 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.392173 4865 scope.go:122] "RemoveContainer" containerID="96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2" Oct 31 16:06:14.393080 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:06:14.393048 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2\": container with ID starting with 96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2 not found: ID does not exist" containerID="96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2" Oct 31 16:06:14.393222 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.393086 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2"} err="failed to get container status \"96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2\": rpc error: code = NotFound desc = could not find container \"96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2\": container with ID starting with 96675e2c9ba5ea1a6923cb0e4af0666afd7c66b77c1af9eb1c67f46c2733d9f2 not found: ID does not exist" Oct 31 16:06:14.455710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.455687 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 16:06:14.526179 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.526149 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt"] Oct 31 16:06:14.530712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.530688 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-aa864d-7d84c4b7f7-k4zlt"] Oct 31 16:06:14.570832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.570805 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-podinfo\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.570940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.570838 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-kube-api-access-v56cg\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-kube-api-access-v56cg\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.570940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.570873 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istiod-ca-cert\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.570940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.570905 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-socket\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.570940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.570940 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-envoy\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.571151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.570958 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-data\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.571151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.570990 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-token\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-token\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.571151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571078 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-certs\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.571151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571120 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-credential-socket\") pod \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\" (UID: \"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1\") " Oct 31 16:06:14.571409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571312 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-data" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:14.571409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571341 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-socket" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:14.571574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571522 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-certs" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:14.571574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571557 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-credential-socket" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:14.571681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571606 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.571681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571629 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.571756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.571670 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istiod-ca-cert" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:06:14.573389 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.573353 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-envoy" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:14.573521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.573360 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-podinfo" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:06:14.573521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.573418 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-kube-api-access-v56cg" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "kube-api-access-v56cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:14.573605 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.573575 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-token" pod "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" (UID: "48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:14.629455 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.629426 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64"] Oct 31 16:06:14.644376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.644349 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq"] Oct 31 16:06:14.651860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.651835 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2"] Oct 31 16:06:14.672327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.672300 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.672327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.672326 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.672504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.672336 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.672504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.672344 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.672504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.672354 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v56cg\" (UniqueName: \"kubernetes.io/projected/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-kube-api-access-v56cg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.672504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.672365 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.672504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.672373 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:14.698636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.698562 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d"] Oct 31 16:06:14.700744 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.700719 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-aa864d-655666b69f-vlc4d"] Oct 31 16:06:14.752820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.752792 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:14.904445 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:14.904391 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:15.371901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:15.371859 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" Oct 31 16:06:15.371901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:15.371893 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv" event={"ID":"48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1","Type":"ContainerDied","Data":"31f954805b1d9003adda44c4bde7e4b0980c79002936a796dd71651b84664c22"} Oct 31 16:06:15.372423 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:15.371950 4865 scope.go:122] "RemoveContainer" containerID="d1b6aade064ad7c4a53c1d563c4c156415727ada83484d664a2cf14df243a1d9" Oct 31 16:06:15.539851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:15.539819 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv"] Oct 31 16:06:15.547201 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:15.547173 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-aa864d-openshift-default-99f549bcf-hzngv"] Oct 31 16:06:16.434157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:16.434125 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:16.495577 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:16.495549 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" path="/var/lib/kubelet/pods/48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1/volumes" Oct 31 16:06:16.495924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:16.495910 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f1e816-7f06-4c01-9d69-eb51b87787e3" path="/var/lib/kubelet/pods/52f1e816-7f06-4c01-9d69-eb51b87787e3/volumes" Oct 31 16:06:16.496233 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:16.496222 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6951eea9-0a8d-4278-88e0-982426951c8c" path="/var/lib/kubelet/pods/6951eea9-0a8d-4278-88e0-982426951c8c/volumes" Oct 31 16:06:17.489107 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:06:17.489071 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:06:18.984028 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:18.983989 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:19.489324 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:19.489291 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:20.334171 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:20.334123 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:24.178184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178144 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh"] Oct 31 16:06:24.178860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178774 4865 cpu_manager.go:414] "containerMap: removing container" podUID="52f1e816-7f06-4c01-9d69-eb51b87787e3" containerName="payload-pre-processing" Oct 31 16:06:24.178860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178795 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f1e816-7f06-4c01-9d69-eb51b87787e3" containerName="payload-pre-processing" Oct 31 16:06:24.178860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178815 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6951eea9-0a8d-4278-88e0-982426951c8c" containerName="payload-processing" Oct 31 16:06:24.178860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178824 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6951eea9-0a8d-4278-88e0-982426951c8c" containerName="payload-processing" Oct 31 16:06:24.178860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178841 4865 cpu_manager.go:414] "containerMap: removing container" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" Oct 31 16:06:24.178860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178848 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" Oct 31 16:06:24.179210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178877 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9f789656-052e-4c05-acef-cca7eaf9d108" containerName="revoke-keys" Oct 31 16:06:24.179210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178884 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f789656-052e-4c05-acef-cca7eaf9d108" containerName="revoke-keys" Oct 31 16:06:24.179210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.178997 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f789656-052e-4c05-acef-cca7eaf9d108" containerName="revoke-keys" Oct 31 16:06:24.179210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.179011 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" Oct 31 16:06:24.179210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.179025 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f1e816-7f06-4c01-9d69-eb51b87787e3" containerName="payload-pre-processing" Oct 31 16:06:24.179210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.179036 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6951eea9-0a8d-4278-88e0-982426951c8c" containerName="payload-processing" Oct 31 16:06:24.183385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.183364 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.202310 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.202282 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:24.288215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.288187 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:06:24.288434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.288221 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:06:24.288434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.288240 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:06:24.288434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.288198 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:06:24.291057 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.291030 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh"] Oct 31 16:06:24.323265 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.323238 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:24.373673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.373640 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.373798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.373704 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.373865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.373841 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-kube-api-access-ch7kx\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.475308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.475272 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.475497 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.475326 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.475497 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.475363 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-kube-api-access-ch7kx\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.475942 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.475920 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.477820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.477795 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.534048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.534015 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-kube-api-access-ch7kx\") pod \"maas-api-revoke-sub-161d3702-slxgh\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.796727 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.796648 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:24.945795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:24.945732 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:25.171617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:25.171593 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh"] Oct 31 16:06:25.172153 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:06:25.172127 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc91dd7_eae5_4a7a_bb8b_c3255957c449.slice/crio-0e3145ffdc0d712e6e56d060cfde94672b304e204cf31ec26d56f2a2742cfca7 WatchSource:0}: Error finding container 0e3145ffdc0d712e6e56d060cfde94672b304e204cf31ec26d56f2a2742cfca7: Status 404 returned error can't find the container with id 0e3145ffdc0d712e6e56d060cfde94672b304e204cf31ec26d56f2a2742cfca7 Oct 31 16:06:25.432391 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:25.432323 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" event={"ID":"0cc91dd7-eae5-4a7a-bb8b-c3255957c449","Type":"ContainerStarted","Data":"0e3145ffdc0d712e6e56d060cfde94672b304e204cf31ec26d56f2a2742cfca7"} Oct 31 16:06:25.742536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:25.742506 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:25.863596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:25.863565 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:26.440980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:26.440885 4865 generic.go:356] "Generic (PLEG): container finished" podID="0cc91dd7-eae5-4a7a-bb8b-c3255957c449" containerID="224525d0fec23233b3dfe7bb495b629c02019e3193b2b0b55f5b64ecf142c25e" exitCode=0 Oct 31 16:06:26.440980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:26.440973 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" event={"ID":"0cc91dd7-eae5-4a7a-bb8b-c3255957c449","Type":"ContainerDied","Data":"224525d0fec23233b3dfe7bb495b629c02019e3193b2b0b55f5b64ecf142c25e"} Oct 31 16:06:27.575259 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.575237 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:27.606703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.606677 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-openshift-ca\") pod \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " Oct 31 16:06:27.606858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.606723 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-xks-ca\") pod \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " Oct 31 16:06:27.607017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.606994 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-openshift-ca" pod "0cc91dd7-eae5-4a7a-bb8b-c3255957c449" (UID: "0cc91dd7-eae5-4a7a-bb8b-c3255957c449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:06:27.607225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.607196 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:27.609210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.609190 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-xks-ca" pod "0cc91dd7-eae5-4a7a-bb8b-c3255957c449" (UID: "0cc91dd7-eae5-4a7a-bb8b-c3255957c449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:06:27.707624 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.707599 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-kube-api-access-ch7kx\") pod \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\" (UID: \"0cc91dd7-eae5-4a7a-bb8b-c3255957c449\") " Oct 31 16:06:27.707958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.707937 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:27.709908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.709885 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-kube-api-access-ch7kx" pod "0cc91dd7-eae5-4a7a-bb8b-c3255957c449" (UID: "0cc91dd7-eae5-4a7a-bb8b-c3255957c449"). InnerVolumeSpecName "kube-api-access-ch7kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:27.809167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:27.809137 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7kx\" (UniqueName: \"kubernetes.io/projected/0cc91dd7-eae5-4a7a-bb8b-c3255957c449-kube-api-access-ch7kx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:28.457942 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:28.457903 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" event={"ID":"0cc91dd7-eae5-4a7a-bb8b-c3255957c449","Type":"ContainerDied","Data":"0e3145ffdc0d712e6e56d060cfde94672b304e204cf31ec26d56f2a2742cfca7"} Oct 31 16:06:28.458165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:28.458001 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3145ffdc0d712e6e56d060cfde94672b304e204cf31ec26d56f2a2742cfca7" Oct 31 16:06:28.458165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:28.457922 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh" Oct 31 16:06:29.199462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.199410 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:29.652850 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.652797 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" containerID="cri-o://052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e" gracePeriod=45 Oct 31 16:06:29.668528 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.668492 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" containerID="cri-o://17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e" gracePeriod=45 Oct 31 16:06:29.671591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.671509 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" containerID="cri-o://bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607" gracePeriod=45 Oct 31 16:06:29.678043 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.678014 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Oct 31 16:06:29.690407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.690373 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused" Oct 31 16:06:29.766569 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.766227 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:06:29.925404 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.925370 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 16:06:29.927331 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927301 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-home\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-home\") pod \"62479a79-197f-4a3f-9c53-bb3e9741fb86\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " Oct 31 16:06:29.927498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927358 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62479a79-197f-4a3f-9c53-bb3e9741fb86-kube-api-access-4zm4n\" (UniqueName: \"kubernetes.io/projected/62479a79-197f-4a3f-9c53-bb3e9741fb86-kube-api-access-4zm4n\") pod \"62479a79-197f-4a3f-9c53-bb3e9741fb86\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " Oct 31 16:06:29.927498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927439 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-model-cache\") pod \"62479a79-197f-4a3f-9c53-bb3e9741fb86\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " Oct 31 16:06:29.927629 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927524 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62479a79-197f-4a3f-9c53-bb3e9741fb86-tls-certs\" (UniqueName: \"kubernetes.io/secret/62479a79-197f-4a3f-9c53-bb3e9741fb86-tls-certs\") pod \"62479a79-197f-4a3f-9c53-bb3e9741fb86\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " Oct 31 16:06:29.927629 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927559 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-dshm\") pod \"62479a79-197f-4a3f-9c53-bb3e9741fb86\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " Oct 31 16:06:29.927629 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927578 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-home" pod "62479a79-197f-4a3f-9c53-bb3e9741fb86" (UID: "62479a79-197f-4a3f-9c53-bb3e9741fb86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:29.927780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927655 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-tmp-dir\") pod \"62479a79-197f-4a3f-9c53-bb3e9741fb86\" (UID: \"62479a79-197f-4a3f-9c53-bb3e9741fb86\") " Oct 31 16:06:29.927940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.927913 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-model-cache" pod "62479a79-197f-4a3f-9c53-bb3e9741fb86" (UID: "62479a79-197f-4a3f-9c53-bb3e9741fb86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:29.928133 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.928110 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:29.928250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.928143 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:29.928250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.928111 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-tmp-dir" pod "62479a79-197f-4a3f-9c53-bb3e9741fb86" (UID: "62479a79-197f-4a3f-9c53-bb3e9741fb86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:29.931063 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.931036 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-dshm" pod "62479a79-197f-4a3f-9c53-bb3e9741fb86" (UID: "62479a79-197f-4a3f-9c53-bb3e9741fb86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:29.931185 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.931165 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62479a79-197f-4a3f-9c53-bb3e9741fb86-kube-api-access-4zm4n" pod "62479a79-197f-4a3f-9c53-bb3e9741fb86" (UID: "62479a79-197f-4a3f-9c53-bb3e9741fb86"). InnerVolumeSpecName "kube-api-access-4zm4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:29.931254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.931197 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62479a79-197f-4a3f-9c53-bb3e9741fb86-tls-certs" pod "62479a79-197f-4a3f-9c53-bb3e9741fb86" (UID: "62479a79-197f-4a3f-9c53-bb3e9741fb86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:06:29.951478 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.951447 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 16:06:29.955994 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:29.955972 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 16:06:30.028525 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028492 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-dshm\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-dshm\") pod \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " Oct 31 16:06:30.028735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028538 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-kube-api-access-9l8hr\" (UniqueName: \"kubernetes.io/projected/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-kube-api-access-9l8hr\") pod \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " Oct 31 16:06:30.028735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028565 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tmp-dir\") pod \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " Oct 31 16:06:30.028735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028589 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-home\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-home\") pod \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " Oct 31 16:06:30.028735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028659 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tmp-dir\") pod \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " Oct 31 16:06:30.028735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028697 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-home\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-home\") pod \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " Oct 31 16:06:30.029015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028740 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-model-cache\") pod \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " Oct 31 16:06:30.029015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028802 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-model-cache\") pod \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " Oct 31 16:06:30.029015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028854 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tls-certs\" (UniqueName: \"kubernetes.io/secret/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tls-certs\") pod \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " Oct 31 16:06:30.029015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028892 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tls-certs\") pod \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " Oct 31 16:06:30.029015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028938 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511b6f1e-2b64-4b95-8d55-6f7523dcd763-kube-api-access-tzbk6\" (UniqueName: \"kubernetes.io/projected/511b6f1e-2b64-4b95-8d55-6f7523dcd763-kube-api-access-tzbk6\") pod \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\" (UID: \"511b6f1e-2b64-4b95-8d55-6f7523dcd763\") " Oct 31 16:06:30.029015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028983 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-dshm\") pod \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\" (UID: \"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc\") " Oct 31 16:06:30.029320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029302 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62479a79-197f-4a3f-9c53-bb3e9741fb86-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.029381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029325 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.029381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029339 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62479a79-197f-4a3f-9c53-bb3e9741fb86-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.029381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029355 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zm4n\" (UniqueName: \"kubernetes.io/projected/62479a79-197f-4a3f-9c53-bb3e9741fb86-kube-api-access-4zm4n\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.029580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028856 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tmp-dir" pod "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" (UID: "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.029580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.028933 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-home" pod "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" (UID: "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.029580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029060 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-model-cache" pod "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" (UID: "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.029580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029198 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tmp-dir" pod "511b6f1e-2b64-4b95-8d55-6f7523dcd763" (UID: "511b6f1e-2b64-4b95-8d55-6f7523dcd763"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.029580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029286 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-home" pod "511b6f1e-2b64-4b95-8d55-6f7523dcd763" (UID: "511b6f1e-2b64-4b95-8d55-6f7523dcd763"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.029580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.029440 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-model-cache" pod "511b6f1e-2b64-4b95-8d55-6f7523dcd763" (UID: "511b6f1e-2b64-4b95-8d55-6f7523dcd763"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.031732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.031638 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-dshm" pod "511b6f1e-2b64-4b95-8d55-6f7523dcd763" (UID: "511b6f1e-2b64-4b95-8d55-6f7523dcd763"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.034512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.034431 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tls-certs" pod "511b6f1e-2b64-4b95-8d55-6f7523dcd763" (UID: "511b6f1e-2b64-4b95-8d55-6f7523dcd763"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:06:30.034512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.034433 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tls-certs" pod "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" (UID: "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:06:30.034512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.034485 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-dshm" pod "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" (UID: "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:30.034512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.034505 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-kube-api-access-9l8hr" pod "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" (UID: "ecb29fd0-3b02-45f9-87b9-b919cf25d4bc"). InnerVolumeSpecName "kube-api-access-9l8hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:30.034512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.034491 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511b6f1e-2b64-4b95-8d55-6f7523dcd763-kube-api-access-tzbk6" pod "511b6f1e-2b64-4b95-8d55-6f7523dcd763" (UID: "511b6f1e-2b64-4b95-8d55-6f7523dcd763"). InnerVolumeSpecName "kube-api-access-tzbk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:30.130167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130134 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l8hr\" (UniqueName: \"kubernetes.io/projected/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-kube-api-access-9l8hr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130165 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130178 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130189 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130200 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130210 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130223 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130235 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511b6f1e-2b64-4b95-8d55-6f7523dcd763-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130248 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130268 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzbk6\" (UniqueName: \"kubernetes.io/projected/511b6f1e-2b64-4b95-8d55-6f7523dcd763-kube-api-access-tzbk6\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130279 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.130414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.130290 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511b6f1e-2b64-4b95-8d55-6f7523dcd763-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:30.471219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.471185 4865 generic.go:356] "Generic (PLEG): container finished" podID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerID="052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e" exitCode=0 Oct 31 16:06:30.471747 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.471233 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" Oct 31 16:06:30.471747 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.471266 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" event={"ID":"62479a79-197f-4a3f-9c53-bb3e9741fb86","Type":"ContainerDied","Data":"052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e"} Oct 31 16:06:30.471747 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.471316 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64" event={"ID":"62479a79-197f-4a3f-9c53-bb3e9741fb86","Type":"ContainerDied","Data":"d85e881bc9f8e28235a2145093203af543c2320253f417d85e734739df5899b7"} Oct 31 16:06:30.471747 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.471337 4865 scope.go:122] "RemoveContainer" containerID="052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e" Oct 31 16:06:30.473388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.473015 4865 generic.go:356] "Generic (PLEG): container finished" podID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerID="17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e" exitCode=0 Oct 31 16:06:30.473388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.473042 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" event={"ID":"511b6f1e-2b64-4b95-8d55-6f7523dcd763","Type":"ContainerDied","Data":"17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e"} Oct 31 16:06:30.473388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.473067 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" Oct 31 16:06:30.473388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.473096 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq" event={"ID":"511b6f1e-2b64-4b95-8d55-6f7523dcd763","Type":"ContainerDied","Data":"9919d43ca5e30ffce0911905f83a7b0e661fcec78e8531212976bd5e182dd503"} Oct 31 16:06:30.474893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.474855 4865 generic.go:356] "Generic (PLEG): container finished" podID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerID="bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607" exitCode=0 Oct 31 16:06:30.475048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.474900 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" event={"ID":"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc","Type":"ContainerDied","Data":"bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607"} Oct 31 16:06:30.475048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.474919 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" Oct 31 16:06:30.475048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.474938 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2" event={"ID":"ecb29fd0-3b02-45f9-87b9-b919cf25d4bc","Type":"ContainerDied","Data":"02f0497581ce0a319df5240f46addd7962add5f12b58ee6b0b61885260248423"} Oct 31 16:06:30.483985 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.483964 4865 scope.go:122] "RemoveContainer" containerID="052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e" Oct 31 16:06:30.484338 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:06:30.484311 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e\": container with ID starting with 052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e not found: ID does not exist" containerID="052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e" Oct 31 16:06:30.484454 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.484347 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e"} err="failed to get container status \"052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e\": rpc error: code = NotFound desc = could not find container \"052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e\": container with ID starting with 052ffaf994d1a3f10c3378043ca705b894f595f333d6efa2b07b187a97a1e04e not found: ID does not exist" Oct 31 16:06:30.484454 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.484365 4865 scope.go:122] "RemoveContainer" containerID="17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e" Oct 31 16:06:30.493660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.493645 4865 scope.go:122] "RemoveContainer" containerID="17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e" Oct 31 16:06:30.493959 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:06:30.493939 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e\": container with ID starting with 17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e not found: ID does not exist" containerID="17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e" Oct 31 16:06:30.494028 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.493966 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e"} err="failed to get container status \"17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e\": rpc error: code = NotFound desc = could not find container \"17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e\": container with ID starting with 17e49f4d1aed27e3e4e37360de77eb3ad7704d5ca3d318dba710fc623c01e80e not found: ID does not exist" Oct 31 16:06:30.494028 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.493983 4865 scope.go:122] "RemoveContainer" containerID="bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607" Oct 31 16:06:30.502519 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.502497 4865 scope.go:122] "RemoveContainer" containerID="bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607" Oct 31 16:06:30.502793 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:06:30.502772 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607\": container with ID starting with bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607 not found: ID does not exist" containerID="bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607" Oct 31 16:06:30.502888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.502803 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607"} err="failed to get container status \"bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607\": rpc error: code = NotFound desc = could not find container \"bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607\": container with ID starting with bdd5c2215ce60f46c6df922634d7892fe549d4eec5e30a4183d12b4e30202607 not found: ID does not exist" Oct 31 16:06:30.593877 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.593838 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:30.642283 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.642248 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2"] Oct 31 16:06:30.645815 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.645783 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/simulator-w1-fb2348-kserve-784b4785fc-v4dd2"] Oct 31 16:06:30.756670 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.756579 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64"] Oct 31 16:06:30.758969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.758944 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-2-w1-fb2348-kserve-64cbd7878d-ldz64"] Oct 31 16:06:30.870338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.870305 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq"] Oct 31 16:06:30.875874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:30.875851 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/premium-w1-fb2348-kserve-9bd649868-jphtq"] Oct 31 16:06:32.494524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:32.494495 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" path="/var/lib/kubelet/pods/511b6f1e-2b64-4b95-8d55-6f7523dcd763/volumes" Oct 31 16:06:32.494932 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:32.494919 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" path="/var/lib/kubelet/pods/62479a79-197f-4a3f-9c53-bb3e9741fb86/volumes" Oct 31 16:06:32.495288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:32.495274 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" path="/var/lib/kubelet/pods/ecb29fd0-3b02-45f9-87b9-b919cf25d4bc/volumes" Oct 31 16:06:33.903863 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:33.903823 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:36.141851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:36.141817 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:38.652065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:38.652027 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:42.658969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:42.658939 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:44.405310 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:44.405275 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:47.488927 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:06:47.488889 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:06:49.095096 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:49.095059 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:50.461577 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:50.461536 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:50.708443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:50.708406 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:51.130742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:51.130707 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:51.332375 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:51.332340 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:53.043739 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:53.043702 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:53.799158 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:53.799119 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:56.356137 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:56.356092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:56.855738 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:56.855703 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:56.872895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:56.872866 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq"] Oct 31 16:06:56.873104 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:56.873069 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" containerID="cri-o://0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d" gracePeriod=30 Oct 31 16:06:58.852818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:58.852780 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:59.100293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:06:59.100248 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:00.117917 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:00.117887 4865 scope.go:122] "RemoveContainer" containerID="47c147679065b0abd35420a5a3653e607f8936cf1dc21676b7de2d253a1b02df" Oct 31 16:07:00.127116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:00.127092 4865 scope.go:122] "RemoveContainer" containerID="aaf1892a1f5751f7a79af36be38cbac7893e5bfb2842168cd3f368c04e4f36f9" Oct 31 16:07:01.927582 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:01.927550 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:07:02.112630 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.112602 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 16:07:02.245815 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.245786 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-token\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-token\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246009 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.245830 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-podinfo\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246009 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.245883 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-kube-api-access-cdpwm\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-kube-api-access-cdpwm\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246009 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.245937 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-credential-socket\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246013 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-envoy\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246047 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-certs\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246073 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ad1b6a-2eab-443b-93a0-885e77829e2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ad1b6a-2eab-443b-93a0-885e77829e2c-istiod-ca-cert\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246109 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-data\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246179 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-socket\") pod \"90ad1b6a-2eab-443b-93a0-885e77829e2c\" (UID: \"90ad1b6a-2eab-443b-93a0-885e77829e2c\") " Oct 31 16:07:02.246568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246539 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-certs" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:02.246703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246674 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-credential-socket" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:02.246861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246841 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-data" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:02.246942 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.246895 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-socket" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:02.247077 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.247053 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ad1b6a-2eab-443b-93a0-885e77829e2c-istiod-ca-cert" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:07:02.248391 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.248363 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-envoy" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:02.248537 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.248365 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-podinfo" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:07:02.248681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.248664 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-token" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:02.248798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.248773 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-kube-api-access-cdpwm" pod "90ad1b6a-2eab-443b-93a0-885e77829e2c" (UID: "90ad1b6a-2eab-443b-93a0-885e77829e2c"). InnerVolumeSpecName "kube-api-access-cdpwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:02.347055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347019 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347049 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ad1b6a-2eab-443b-93a0-885e77829e2c-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347060 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347069 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347078 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347086 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347096 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdpwm\" (UniqueName: \"kubernetes.io/projected/90ad1b6a-2eab-443b-93a0-885e77829e2c-kube-api-access-cdpwm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347105 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.347284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.347113 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ad1b6a-2eab-443b-93a0-885e77829e2c-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:02.657450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.657417 4865 generic.go:356] "Generic (PLEG): container finished" podID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerID="0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d" exitCode=0 Oct 31 16:07:02.657647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.657486 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" event={"ID":"90ad1b6a-2eab-443b-93a0-885e77829e2c","Type":"ContainerDied","Data":"0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d"} Oct 31 16:07:02.657647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.657530 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" event={"ID":"90ad1b6a-2eab-443b-93a0-885e77829e2c","Type":"ContainerDied","Data":"81f927a20f2b30f2fb8ef83575d2b333fe3b96fadd7fce063d21bec314a1780d"} Oct 31 16:07:02.657647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.657551 4865 scope.go:122] "RemoveContainer" containerID="0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d" Oct 31 16:07:02.657647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.657567 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq" Oct 31 16:07:02.667590 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.667569 4865 scope.go:122] "RemoveContainer" containerID="72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523" Oct 31 16:07:02.676260 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.676235 4865 scope.go:122] "RemoveContainer" containerID="0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d" Oct 31 16:07:02.676596 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:02.676567 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d\": container with ID starting with 0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d not found: ID does not exist" containerID="0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d" Oct 31 16:07:02.676682 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.676604 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d"} err="failed to get container status \"0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d\": rpc error: code = NotFound desc = could not find container \"0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d\": container with ID starting with 0ea08072a7e212969cb6b94735cab882e2cc90f0220e66804b1b3c7ceb5a3c5d not found: ID does not exist" Oct 31 16:07:02.676682 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.676622 4865 scope.go:122] "RemoveContainer" containerID="72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523" Oct 31 16:07:02.676895 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:02.676878 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523\": container with ID starting with 72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523 not found: ID does not exist" containerID="72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523" Oct 31 16:07:02.676946 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.676899 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523"} err="failed to get container status \"72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523\": rpc error: code = NotFound desc = could not find container \"72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523\": container with ID starting with 72fa762642e859a313dbe51d26a7d37db46a4c3cd89e6d6482963c6297f4f523 not found: ID does not exist" Oct 31 16:07:02.824944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.824861 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq"] Oct 31 16:07:02.875949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:02.875911 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-md9rq"] Oct 31 16:07:03.502060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:03.502026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:04.495106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:04.495073 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" path="/var/lib/kubelet/pods/90ad1b6a-2eab-443b-93a0-885e77829e2c/volumes" Oct 31 16:07:07.080532 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:07.080493 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:07.212672 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:07.212637 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:08.194648 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:08.194611 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:12.805886 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:12.805857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:12.930615 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:12.930585 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:14.292674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:14.292636 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:17.490369 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:17.490333 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:07:17.684701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:17.684664 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:22.354323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:22.354281 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:25.548090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:25.548053 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:07:27.002809 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:27.002748 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:28.784195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784161 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-0ccc71"] Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784559 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0cc91dd7-eae5-4a7a-bb8b-c3255957c449" containerName="revoke-keys" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784570 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc91dd7-eae5-4a7a-bb8b-c3255957c449" containerName="revoke-keys" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784590 4865 cpu_manager.go:414] "containerMap: removing container" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784596 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784602 4865 cpu_manager.go:414] "containerMap: removing container" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784607 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784617 4865 cpu_manager.go:414] "containerMap: removing container" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784622 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784636 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784641 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784647 4865 cpu_manager.go:414] "containerMap: removing container" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784652 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784659 4865 cpu_manager.go:414] "containerMap: removing container" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" Oct 31 16:07:28.784661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784663 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" Oct 31 16:07:28.785214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784721 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="511b6f1e-2b64-4b95-8d55-6f7523dcd763" containerName="main" Oct 31 16:07:28.785214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784731 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" Oct 31 16:07:28.785214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784739 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb29fd0-3b02-45f9-87b9-b919cf25d4bc" containerName="main" Oct 31 16:07:28.785214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784745 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ad1b6a-2eab-443b-93a0-885e77829e2c" containerName="istio-proxy" Oct 31 16:07:28.785214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784750 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="62479a79-197f-4a3f-9c53-bb3e9741fb86" containerName="main" Oct 31 16:07:28.785214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784757 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d447b6-4d3d-41dd-8a2f-ba4e13cf46c1" containerName="istio-proxy" Oct 31 16:07:28.785214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.784765 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc91dd7-eae5-4a7a-bb8b-c3255957c449" containerName="revoke-keys" Oct 31 16:07:28.787927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.787909 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:28.842197 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.842168 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-0ccc71"] Oct 31 16:07:28.890795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.890765 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6nthq\"" Oct 31 16:07:28.904868 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:28.904842 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccwk8\" (UniqueName: \"kubernetes.io/projected/03edb904-d3f5-4ee9-91f6-8239cc775c7c-kube-api-access-ccwk8\") pod \"test-curl-7579-0ccc71\" (UID: \"03edb904-d3f5-4ee9-91f6-8239cc775c7c\") " pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:29.006010 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:29.005977 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccwk8\" (UniqueName: \"kubernetes.io/projected/03edb904-d3f5-4ee9-91f6-8239cc775c7c-kube-api-access-ccwk8\") pod \"test-curl-7579-0ccc71\" (UID: \"03edb904-d3f5-4ee9-91f6-8239cc775c7c\") " pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:29.063971 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:29.063895 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccwk8\" (UniqueName: \"kubernetes.io/projected/03edb904-d3f5-4ee9-91f6-8239cc775c7c-kube-api-access-ccwk8\") pod \"test-curl-7579-0ccc71\" (UID: \"03edb904-d3f5-4ee9-91f6-8239cc775c7c\") " pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:29.099480 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:29.099440 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:29.469012 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:07:29.468980 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03edb904_d3f5_4ee9_91f6_8239cc775c7c.slice/crio-ea64d52978c6f6068adb26101cb6d385f6f46cb194c5f52dc29c3833a9a27fbb WatchSource:0}: Error finding container ea64d52978c6f6068adb26101cb6d385f6f46cb194c5f52dc29c3833a9a27fbb: Status 404 returned error can't find the container with id ea64d52978c6f6068adb26101cb6d385f6f46cb194c5f52dc29c3833a9a27fbb Oct 31 16:07:29.469453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:29.469406 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-0ccc71"] Oct 31 16:07:29.812532 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:29.812443 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-0ccc71" event={"ID":"03edb904-d3f5-4ee9-91f6-8239cc775c7c","Type":"ContainerStarted","Data":"05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f"} Oct 31 16:07:29.812532 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:29.812516 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-0ccc71" event={"ID":"03edb904-d3f5-4ee9-91f6-8239cc775c7c","Type":"ContainerStarted","Data":"ea64d52978c6f6068adb26101cb6d385f6f46cb194c5f52dc29c3833a9a27fbb"} Oct 31 16:07:29.926882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:29.926831 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-0ccc71" podStartSLOduration=1.926815234 podStartE2EDuration="1.926815234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:29.924895874 +0000 UTC m=+4895.544447286" watchObservedRunningTime="2026-10-31 16:07:29.926815234 +0000 UTC m=+4895.546366644" Oct 31 16:07:31.351439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.351388 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-0ccc71"] Oct 31 16:07:31.354812 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.354787 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-0ccc71"] Oct 31 16:07:31.488318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.488279 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-228a51"] Oct 31 16:07:31.488710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.488695 4865 cpu_manager.go:414] "containerMap: removing container" podUID="03edb904-d3f5-4ee9-91f6-8239cc775c7c" containerName="test-curl-7579-0ccc71" Oct 31 16:07:31.488768 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.488711 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03edb904-d3f5-4ee9-91f6-8239cc775c7c" containerName="test-curl-7579-0ccc71" Oct 31 16:07:31.488818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.488809 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="03edb904-d3f5-4ee9-91f6-8239cc775c7c" containerName="test-curl-7579-0ccc71" Oct 31 16:07:31.492182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.492157 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:31.545397 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.545370 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-228a51"] Oct 31 16:07:31.629835 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.629760 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlx5\" (UniqueName: \"kubernetes.io/projected/615f6663-c96e-44e6-9642-309e0b404f15-kube-api-access-cnlx5\") pod \"test-curl-7579-228a51\" (UID: \"615f6663-c96e-44e6-9642-309e0b404f15\") " pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:31.695514 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.695460 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:31.731193 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.731161 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlx5\" (UniqueName: \"kubernetes.io/projected/615f6663-c96e-44e6-9642-309e0b404f15-kube-api-access-cnlx5\") pod \"test-curl-7579-228a51\" (UID: \"615f6663-c96e-44e6-9642-309e0b404f15\") " pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:31.793040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.793014 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlx5\" (UniqueName: \"kubernetes.io/projected/615f6663-c96e-44e6-9642-309e0b404f15-kube-api-access-cnlx5\") pod \"test-curl-7579-228a51\" (UID: \"615f6663-c96e-44e6-9642-309e0b404f15\") " pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:31.803908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.803877 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:31.823637 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:31.823606 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-0ccc71" podUID="03edb904-d3f5-4ee9-91f6-8239cc775c7c" containerName="test-curl-7579-0ccc71" containerID="cri-o://05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f" gracePeriod=2 Oct 31 16:07:31.878114 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:31.878078 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7579-0ccc71\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="03edb904-d3f5-4ee9-91f6-8239cc775c7c" pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:32.174842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:32.174807 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-228a51"] Oct 31 16:07:32.175616 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:07:32.175590 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod615f6663_c96e_44e6_9642_309e0b404f15.slice/crio-34a899e1ef5a30aaf82ac424c737af30ecc6a13a3e87fedd2ad2e4ab9d936d49 WatchSource:0}: Error finding container 34a899e1ef5a30aaf82ac424c737af30ecc6a13a3e87fedd2ad2e4ab9d936d49: Status 404 returned error can't find the container with id 34a899e1ef5a30aaf82ac424c737af30ecc6a13a3e87fedd2ad2e4ab9d936d49 Oct 31 16:07:32.830333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:32.830294 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-228a51" event={"ID":"615f6663-c96e-44e6-9642-309e0b404f15","Type":"ContainerStarted","Data":"e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538"} Oct 31 16:07:32.830753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:32.830340 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-228a51" event={"ID":"615f6663-c96e-44e6-9642-309e0b404f15","Type":"ContainerStarted","Data":"34a899e1ef5a30aaf82ac424c737af30ecc6a13a3e87fedd2ad2e4ab9d936d49"} Oct 31 16:07:32.945224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:32.945172 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-228a51" podStartSLOduration=1.9451592309999999 podStartE2EDuration="1.945159231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:32.943732746 +0000 UTC m=+4898.563284158" watchObservedRunningTime="2026-10-31 16:07:32.945159231 +0000 UTC m=+4898.564710642" Oct 31 16:07:34.041129 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.041102 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:34.154347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.154260 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03edb904-d3f5-4ee9-91f6-8239cc775c7c-kube-api-access-ccwk8\" (UniqueName: \"kubernetes.io/projected/03edb904-d3f5-4ee9-91f6-8239cc775c7c-kube-api-access-ccwk8\") pod \"03edb904-d3f5-4ee9-91f6-8239cc775c7c\" (UID: \"03edb904-d3f5-4ee9-91f6-8239cc775c7c\") " Oct 31 16:07:34.156421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.156394 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03edb904-d3f5-4ee9-91f6-8239cc775c7c-kube-api-access-ccwk8" pod "03edb904-d3f5-4ee9-91f6-8239cc775c7c" (UID: "03edb904-d3f5-4ee9-91f6-8239cc775c7c"). InnerVolumeSpecName "kube-api-access-ccwk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:34.255627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.255598 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccwk8\" (UniqueName: \"kubernetes.io/projected/03edb904-d3f5-4ee9-91f6-8239cc775c7c-kube-api-access-ccwk8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:34.312804 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.312776 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-228a51"] Oct 31 16:07:34.315804 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.315776 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-228a51"] Oct 31 16:07:34.494453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.494421 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03edb904-d3f5-4ee9-91f6-8239cc775c7c" path="/var/lib/kubelet/pods/03edb904-d3f5-4ee9-91f6-8239cc775c7c/volumes" Oct 31 16:07:34.841842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.841750 4865 generic.go:356] "Generic (PLEG): container finished" podID="03edb904-d3f5-4ee9-91f6-8239cc775c7c" containerID="05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f" exitCode=137 Oct 31 16:07:34.841842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.841793 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-0ccc71" Oct 31 16:07:34.841842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.841837 4865 scope.go:122] "RemoveContainer" containerID="05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f" Oct 31 16:07:34.842153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.841995 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-228a51" podUID="615f6663-c96e-44e6-9642-309e0b404f15" containerName="test-curl-7579-228a51" containerID="cri-o://e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538" gracePeriod=2 Oct 31 16:07:34.850660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.850642 4865 scope.go:122] "RemoveContainer" containerID="05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f" Oct 31 16:07:34.850914 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:34.850899 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f\": container with ID starting with 05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f not found: ID does not exist" containerID="05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f" Oct 31 16:07:34.850960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:34.850925 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f"} err="failed to get container status \"05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f\": rpc error: code = NotFound desc = could not find container \"05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f\": container with ID starting with 05528231e9a85d8710424a1765168e37f794a0b9e03fc19015138f75d444b61f not found: ID does not exist" Oct 31 16:07:35.324109 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.324005 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r"] Oct 31 16:07:35.324667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.324646 4865 cpu_manager.go:414] "containerMap: removing container" podUID="615f6663-c96e-44e6-9642-309e0b404f15" containerName="test-curl-7579-228a51" Oct 31 16:07:35.324755 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.324669 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615f6663-c96e-44e6-9642-309e0b404f15" containerName="test-curl-7579-228a51" Oct 31 16:07:35.324814 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.324773 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="615f6663-c96e-44e6-9642-309e0b404f15" containerName="test-curl-7579-228a51" Oct 31 16:07:35.328801 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.328779 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.383385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.383353 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r"] Oct 31 16:07:35.431703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.431673 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:07:35.431874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.431804 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:07:35.431874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.431858 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:07:35.431972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.431887 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:07:35.465824 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.465798 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99980790-5801-4522-822b-27a52269fff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.465958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.465835 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68s9\" (UniqueName: \"kubernetes.io/projected/99980790-5801-4522-822b-27a52269fff9-kube-api-access-z68s9\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.465958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.465923 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99980790-5801-4522-822b-27a52269fff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.566388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.566352 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99980790-5801-4522-822b-27a52269fff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.566607 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.566411 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68s9\" (UniqueName: \"kubernetes.io/projected/99980790-5801-4522-822b-27a52269fff9-kube-api-access-z68s9\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.566607 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.566501 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99980790-5801-4522-822b-27a52269fff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.567015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.566985 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99980790-5801-4522-822b-27a52269fff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.568963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.568943 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99980790-5801-4522-822b-27a52269fff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.625037 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.624965 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68s9\" (UniqueName: \"kubernetes.io/projected/99980790-5801-4522-822b-27a52269fff9-kube-api-access-z68s9\") pod \"maas-api-revoke-sub-736a0494-wg79r\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:35.641236 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:35.641204 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:36.014060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:36.014032 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r"] Oct 31 16:07:36.014205 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:07:36.014052 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99980790_5801_4522_822b_27a52269fff9.slice/crio-f67abf004bbfdf7789ed522ae70cf2b07464676806a8a7bb18b0fa87b1a9bc7a WatchSource:0}: Error finding container f67abf004bbfdf7789ed522ae70cf2b07464676806a8a7bb18b0fa87b1a9bc7a: Status 404 returned error can't find the container with id f67abf004bbfdf7789ed522ae70cf2b07464676806a8a7bb18b0fa87b1a9bc7a Oct 31 16:07:36.492710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:36.492677 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:36.856210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:36.856181 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" event={"ID":"99980790-5801-4522-822b-27a52269fff9","Type":"ContainerStarted","Data":"1316ae5a3551e9ff55cb19a4f6b4b516636c9f975af61fcce839b90bc6a3ef52"} Oct 31 16:07:36.856353 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:36.856232 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" event={"ID":"99980790-5801-4522-822b-27a52269fff9","Type":"ContainerStarted","Data":"f67abf004bbfdf7789ed522ae70cf2b07464676806a8a7bb18b0fa87b1a9bc7a"} Oct 31 16:07:36.973921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:36.973863 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" podStartSLOduration=1.973849771 podStartE2EDuration="1.973849771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:36.971592535 +0000 UTC m=+4902.591143947" watchObservedRunningTime="2026-10-31 16:07:36.973849771 +0000 UTC m=+4902.593401239" Oct 31 16:07:37.069776 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.069754 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:37.182378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.182305 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615f6663-c96e-44e6-9642-309e0b404f15-kube-api-access-cnlx5\" (UniqueName: \"kubernetes.io/projected/615f6663-c96e-44e6-9642-309e0b404f15-kube-api-access-cnlx5\") pod \"615f6663-c96e-44e6-9642-309e0b404f15\" (UID: \"615f6663-c96e-44e6-9642-309e0b404f15\") " Oct 31 16:07:37.184500 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.184458 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615f6663-c96e-44e6-9642-309e0b404f15-kube-api-access-cnlx5" pod "615f6663-c96e-44e6-9642-309e0b404f15" (UID: "615f6663-c96e-44e6-9642-309e0b404f15"). InnerVolumeSpecName "kube-api-access-cnlx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:37.283060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.283031 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnlx5\" (UniqueName: \"kubernetes.io/projected/615f6663-c96e-44e6-9642-309e0b404f15-kube-api-access-cnlx5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:37.862680 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.862590 4865 generic.go:356] "Generic (PLEG): container finished" podID="615f6663-c96e-44e6-9642-309e0b404f15" containerID="e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538" exitCode=137 Oct 31 16:07:37.862680 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.862636 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:37.863230 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.862687 4865 scope.go:122] "RemoveContainer" containerID="e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538" Oct 31 16:07:37.864536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.864512 4865 generic.go:356] "Generic (PLEG): container finished" podID="99980790-5801-4522-822b-27a52269fff9" containerID="1316ae5a3551e9ff55cb19a4f6b4b516636c9f975af61fcce839b90bc6a3ef52" exitCode=0 Oct 31 16:07:37.864663 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.864591 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" event={"ID":"99980790-5801-4522-822b-27a52269fff9","Type":"ContainerDied","Data":"1316ae5a3551e9ff55cb19a4f6b4b516636c9f975af61fcce839b90bc6a3ef52"} Oct 31 16:07:37.872212 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.872192 4865 scope.go:122] "RemoveContainer" containerID="e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538" Oct 31 16:07:37.872460 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:37.872443 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538\": container with ID starting with e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538 not found: ID does not exist" containerID="e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538" Oct 31 16:07:37.872570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:37.872491 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538"} err="failed to get container status \"e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538\": rpc error: code = NotFound desc = could not find container \"e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538\": container with ID starting with e7fc5ec21cba55bb98cb7da17de7de00c043540ce3efe5cebf99633675daa538 not found: ID does not exist" Oct 31 16:07:38.031293 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:38.031255 4865 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7579-228a51\" is forbidden: User \"system:node:ip-10-0-131-60.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-60.us-west-1.compute.internal' and this object" podUID="615f6663-c96e-44e6-9642-309e0b404f15" pod="opendatahub/test-curl-7579-228a51" Oct 31 16:07:38.045335 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:38.045303 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl"] Oct 31 16:07:38.048114 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:38.048095 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3726ba4a-69djl"] Oct 31 16:07:38.495712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:38.495679 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c771321-426c-4a86-9f12-6ccf496115a8" path="/var/lib/kubelet/pods/1c771321-426c-4a86-9f12-6ccf496115a8/volumes" Oct 31 16:07:38.496081 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:38.496066 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615f6663-c96e-44e6-9642-309e0b404f15" path="/var/lib/kubelet/pods/615f6663-c96e-44e6-9642-309e0b404f15/volumes" Oct 31 16:07:39.012828 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.012798 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:39.099609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.099575 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99980790-5801-4522-822b-27a52269fff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99980790-5801-4522-822b-27a52269fff9-maas-api-xks-ca\") pod \"99980790-5801-4522-822b-27a52269fff9\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " Oct 31 16:07:39.099794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.099643 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99980790-5801-4522-822b-27a52269fff9-kube-api-access-z68s9\" (UniqueName: \"kubernetes.io/projected/99980790-5801-4522-822b-27a52269fff9-kube-api-access-z68s9\") pod \"99980790-5801-4522-822b-27a52269fff9\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " Oct 31 16:07:39.099848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.099814 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99980790-5801-4522-822b-27a52269fff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99980790-5801-4522-822b-27a52269fff9-maas-api-openshift-ca\") pod \"99980790-5801-4522-822b-27a52269fff9\" (UID: \"99980790-5801-4522-822b-27a52269fff9\") " Oct 31 16:07:39.100153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.100130 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99980790-5801-4522-822b-27a52269fff9-maas-api-openshift-ca" pod "99980790-5801-4522-822b-27a52269fff9" (UID: "99980790-5801-4522-822b-27a52269fff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:07:39.101971 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.101943 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99980790-5801-4522-822b-27a52269fff9-maas-api-xks-ca" pod "99980790-5801-4522-822b-27a52269fff9" (UID: "99980790-5801-4522-822b-27a52269fff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:07:39.102083 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.101969 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99980790-5801-4522-822b-27a52269fff9-kube-api-access-z68s9" pod "99980790-5801-4522-822b-27a52269fff9" (UID: "99980790-5801-4522-822b-27a52269fff9"). InnerVolumeSpecName "kube-api-access-z68s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:39.201107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.201028 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99980790-5801-4522-822b-27a52269fff9-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:39.201107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.201054 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99980790-5801-4522-822b-27a52269fff9-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:39.201107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.201063 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68s9\" (UniqueName: \"kubernetes.io/projected/99980790-5801-4522-822b-27a52269fff9-kube-api-access-z68s9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:39.878280 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.878242 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" Oct 31 16:07:39.878445 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.878277 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r" event={"ID":"99980790-5801-4522-822b-27a52269fff9","Type":"ContainerDied","Data":"f67abf004bbfdf7789ed522ae70cf2b07464676806a8a7bb18b0fa87b1a9bc7a"} Oct 31 16:07:39.878445 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:39.878327 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f67abf004bbfdf7789ed522ae70cf2b07464676806a8a7bb18b0fa87b1a9bc7a" Oct 31 16:07:41.195582 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:41.195549 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:45.912777 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:45.912744 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:46.085804 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.085764 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5"] Oct 31 16:07:46.086235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.086216 4865 cpu_manager.go:414] "containerMap: removing container" podUID="99980790-5801-4522-822b-27a52269fff9" containerName="revoke-keys" Oct 31 16:07:46.086332 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.086237 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99980790-5801-4522-822b-27a52269fff9" containerName="revoke-keys" Oct 31 16:07:46.086390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.086371 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="99980790-5801-4522-822b-27a52269fff9" containerName="revoke-keys" Oct 31 16:07:46.089530 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.089509 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.143957 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.143924 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:07:46.143957 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.143946 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:07:46.145647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.145626 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5"] Oct 31 16:07:46.194289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.194211 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:07:46.194289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.194214 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:07:46.263883 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.263843 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvq6b\" (UniqueName: \"kubernetes.io/projected/e864631b-8ce8-4890-90dd-4965656c9333-kube-api-access-pvq6b\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.264064 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.263907 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.264064 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.264027 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.364553 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.364521 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.364771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.364570 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.364771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.364633 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvq6b\" (UniqueName: \"kubernetes.io/projected/e864631b-8ce8-4890-90dd-4965656c9333-kube-api-access-pvq6b\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.365248 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.365218 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.367103 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.367082 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.422347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.422312 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvq6b\" (UniqueName: \"kubernetes.io/projected/e864631b-8ce8-4890-90dd-4965656c9333-kube-api-access-pvq6b\") pod \"maas-api-revoke-sub-767911c0-skrk5\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:46.702460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:46.702426 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:47.072649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:47.072621 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5"] Oct 31 16:07:47.073907 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:07:47.073873 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode864631b_8ce8_4890_90dd_4965656c9333.slice/crio-ee08ed9819a5535e7f607b369a9a38ef6b882aa6fe0f7821118856cb320f2310 WatchSource:0}: Error finding container ee08ed9819a5535e7f607b369a9a38ef6b882aa6fe0f7821118856cb320f2310: Status 404 returned error can't find the container with id ee08ed9819a5535e7f607b369a9a38ef6b882aa6fe0f7821118856cb320f2310 Oct 31 16:07:47.488950 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:07:47.488911 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:07:47.922645 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:47.922607 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" event={"ID":"e864631b-8ce8-4890-90dd-4965656c9333","Type":"ContainerStarted","Data":"de9cb6dc5718f7b9f5441cfeb324e5eed75e9cdef2137e17d6024264174101db"} Oct 31 16:07:47.922866 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:47.922665 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" event={"ID":"e864631b-8ce8-4890-90dd-4965656c9333","Type":"ContainerStarted","Data":"ee08ed9819a5535e7f607b369a9a38ef6b882aa6fe0f7821118856cb320f2310"} Oct 31 16:07:48.039359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:48.039307 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" podStartSLOduration=2.039291997 podStartE2EDuration="2.039291997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:48.038126641 +0000 UTC m=+4913.657678052" watchObservedRunningTime="2026-10-31 16:07:48.039291997 +0000 UTC m=+4913.658843445" Oct 31 16:07:48.929064 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:48.929019 4865 generic.go:356] "Generic (PLEG): container finished" podID="e864631b-8ce8-4890-90dd-4965656c9333" containerID="de9cb6dc5718f7b9f5441cfeb324e5eed75e9cdef2137e17d6024264174101db" exitCode=0 Oct 31 16:07:48.929586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:48.929101 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" event={"ID":"e864631b-8ce8-4890-90dd-4965656c9333","Type":"ContainerDied","Data":"de9cb6dc5718f7b9f5441cfeb324e5eed75e9cdef2137e17d6024264174101db"} Oct 31 16:07:49.046608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:49.046572 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2"] Oct 31 16:07:49.051275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:49.051250 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e721021a-v9gt2"] Oct 31 16:07:50.064090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.064062 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:50.203859 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.203781 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-xks-ca\") pod \"e864631b-8ce8-4890-90dd-4965656c9333\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " Oct 31 16:07:50.204033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.203884 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-openshift-ca\") pod \"e864631b-8ce8-4890-90dd-4965656c9333\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " Oct 31 16:07:50.204033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.203955 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e864631b-8ce8-4890-90dd-4965656c9333-kube-api-access-pvq6b\" (UniqueName: \"kubernetes.io/projected/e864631b-8ce8-4890-90dd-4965656c9333-kube-api-access-pvq6b\") pod \"e864631b-8ce8-4890-90dd-4965656c9333\" (UID: \"e864631b-8ce8-4890-90dd-4965656c9333\") " Oct 31 16:07:50.204298 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.204263 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-openshift-ca" pod "e864631b-8ce8-4890-90dd-4965656c9333" (UID: "e864631b-8ce8-4890-90dd-4965656c9333"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:07:50.206191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.206162 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-xks-ca" pod "e864631b-8ce8-4890-90dd-4965656c9333" (UID: "e864631b-8ce8-4890-90dd-4965656c9333"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:07:50.206307 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.206286 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e864631b-8ce8-4890-90dd-4965656c9333-kube-api-access-pvq6b" pod "e864631b-8ce8-4890-90dd-4965656c9333" (UID: "e864631b-8ce8-4890-90dd-4965656c9333"). InnerVolumeSpecName "kube-api-access-pvq6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:50.305206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.305168 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:50.305206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.305201 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvq6b\" (UniqueName: \"kubernetes.io/projected/e864631b-8ce8-4890-90dd-4965656c9333-kube-api-access-pvq6b\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:50.305206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.305215 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864631b-8ce8-4890-90dd-4965656c9333-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:50.498882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.498841 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012106d2-3d26-44c9-8df2-d461d1307801" path="/var/lib/kubelet/pods/012106d2-3d26-44c9-8df2-d461d1307801/volumes" Oct 31 16:07:50.590949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.590908 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:50.942144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.942065 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" Oct 31 16:07:50.942144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.942075 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5" event={"ID":"e864631b-8ce8-4890-90dd-4965656c9333","Type":"ContainerDied","Data":"ee08ed9819a5535e7f607b369a9a38ef6b882aa6fe0f7821118856cb320f2310"} Oct 31 16:07:50.942144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:50.942126 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee08ed9819a5535e7f607b369a9a38ef6b882aa6fe0f7821118856cb320f2310" Oct 31 16:07:51.051584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:51.051547 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:07:55.289036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:55.289003 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:57.674879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.674846 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc"] Oct 31 16:07:57.675314 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.675222 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e864631b-8ce8-4890-90dd-4965656c9333" containerName="revoke-keys" Oct 31 16:07:57.675314 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.675233 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e864631b-8ce8-4890-90dd-4965656c9333" containerName="revoke-keys" Oct 31 16:07:57.675418 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.675318 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e864631b-8ce8-4890-90dd-4965656c9333" containerName="revoke-keys" Oct 31 16:07:57.678770 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.678750 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.734580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.734545 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc"] Oct 31 16:07:57.778414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.778382 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6hr\" (UniqueName: \"kubernetes.io/projected/ae2245fb-0f17-4c1c-92ad-fa23be984423-kube-api-access-pf6hr\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.778594 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.778514 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.778594 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.778556 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.782394 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.782376 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:07:57.782394 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.782378 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:07:57.782589 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.782396 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:07:57.783839 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.783821 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:07:57.879820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.879787 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.879956 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.879842 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.879956 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.879900 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6hr\" (UniqueName: \"kubernetes.io/projected/ae2245fb-0f17-4c1c-92ad-fa23be984423-kube-api-access-pf6hr\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.880462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.880439 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.882367 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.882345 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.940101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.940044 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6hr\" (UniqueName: \"kubernetes.io/projected/ae2245fb-0f17-4c1c-92ad-fa23be984423-kube-api-access-pf6hr\") pod \"maas-api-revoke-sub-8aa6d920-svwnc\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:57.991433 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:57.991410 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:07:58.366452 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:58.366416 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc"] Oct 31 16:07:58.367773 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:07:58.367745 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2245fb_0f17_4c1c_92ad_fa23be984423.slice/crio-d291f4679e78241a5b29f608f4c371b625b193bd4acabf61eb52772056e4e433 WatchSource:0}: Error finding container d291f4679e78241a5b29f608f4c371b625b193bd4acabf61eb52772056e4e433: Status 404 returned error can't find the container with id d291f4679e78241a5b29f608f4c371b625b193bd4acabf61eb52772056e4e433 Oct 31 16:07:58.989048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:58.989012 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" event={"ID":"ae2245fb-0f17-4c1c-92ad-fa23be984423","Type":"ContainerStarted","Data":"c1c671fa3e8c82517b73bd0fc7057da675580cf4ca2fde6124d0d7b799288276"} Oct 31 16:07:58.989543 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:58.989070 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" event={"ID":"ae2245fb-0f17-4c1c-92ad-fa23be984423","Type":"ContainerStarted","Data":"d291f4679e78241a5b29f608f4c371b625b193bd4acabf61eb52772056e4e433"} Oct 31 16:07:59.107882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:59.107780 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" podStartSLOduration=2.1077671159999998 podStartE2EDuration="2.107767116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:59.105208464 +0000 UTC m=+4924.724759876" watchObservedRunningTime="2026-10-31 16:07:59.107767116 +0000 UTC m=+4924.727318567" Oct 31 16:07:59.991969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:59.991910 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:59.996320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:59.996290 4865 generic.go:356] "Generic (PLEG): container finished" podID="ae2245fb-0f17-4c1c-92ad-fa23be984423" containerID="c1c671fa3e8c82517b73bd0fc7057da675580cf4ca2fde6124d0d7b799288276" exitCode=0 Oct 31 16:07:59.996530 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:07:59.996396 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" event={"ID":"ae2245fb-0f17-4c1c-92ad-fa23be984423","Type":"ContainerDied","Data":"c1c671fa3e8c82517b73bd0fc7057da675580cf4ca2fde6124d0d7b799288276"} Oct 31 16:08:01.143261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.143231 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:08:01.212054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.212028 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-openshift-ca\") pod \"ae2245fb-0f17-4c1c-92ad-fa23be984423\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " Oct 31 16:08:01.212226 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.212061 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2245fb-0f17-4c1c-92ad-fa23be984423-kube-api-access-pf6hr\" (UniqueName: \"kubernetes.io/projected/ae2245fb-0f17-4c1c-92ad-fa23be984423-kube-api-access-pf6hr\") pod \"ae2245fb-0f17-4c1c-92ad-fa23be984423\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " Oct 31 16:08:01.212226 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.212161 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-xks-ca\") pod \"ae2245fb-0f17-4c1c-92ad-fa23be984423\" (UID: \"ae2245fb-0f17-4c1c-92ad-fa23be984423\") " Oct 31 16:08:01.212433 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.212409 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-openshift-ca" pod "ae2245fb-0f17-4c1c-92ad-fa23be984423" (UID: "ae2245fb-0f17-4c1c-92ad-fa23be984423"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:08:01.214274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.214242 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-xks-ca" pod "ae2245fb-0f17-4c1c-92ad-fa23be984423" (UID: "ae2245fb-0f17-4c1c-92ad-fa23be984423"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:08:01.214495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.214332 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2245fb-0f17-4c1c-92ad-fa23be984423-kube-api-access-pf6hr" pod "ae2245fb-0f17-4c1c-92ad-fa23be984423" (UID: "ae2245fb-0f17-4c1c-92ad-fa23be984423"). InnerVolumeSpecName "kube-api-access-pf6hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:08:01.313598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.313517 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:01.313598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.313551 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2245fb-0f17-4c1c-92ad-fa23be984423-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:01.313598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:01.313565 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf6hr\" (UniqueName: \"kubernetes.io/projected/ae2245fb-0f17-4c1c-92ad-fa23be984423-kube-api-access-pf6hr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:02.008250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:02.008218 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" Oct 31 16:08:02.008250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:02.008241 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc" event={"ID":"ae2245fb-0f17-4c1c-92ad-fa23be984423","Type":"ContainerDied","Data":"d291f4679e78241a5b29f608f4c371b625b193bd4acabf61eb52772056e4e433"} Oct 31 16:08:02.008498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:02.008290 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d291f4679e78241a5b29f608f4c371b625b193bd4acabf61eb52772056e4e433" Oct 31 16:08:04.698328 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:04.698298 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:07.496141 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:07.496102 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:08:08.621903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.621864 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8"] Oct 31 16:08:08.622326 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.622315 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ae2245fb-0f17-4c1c-92ad-fa23be984423" containerName="revoke-keys" Oct 31 16:08:08.622373 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.622328 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2245fb-0f17-4c1c-92ad-fa23be984423" containerName="revoke-keys" Oct 31 16:08:08.622426 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.622408 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2245fb-0f17-4c1c-92ad-fa23be984423" containerName="revoke-keys" Oct 31 16:08:08.625726 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.625703 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.682350 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.682276 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8"] Oct 31 16:08:08.686991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.686958 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.687179 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.687002 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.687179 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.687034 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8sj\" (UniqueName: \"kubernetes.io/projected/84eb97d5-3053-4003-98b2-7abaeee1fb64-kube-api-access-4l8sj\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.730374 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.730341 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:08:08.730374 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.730364 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:08:08.730642 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.730533 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:08:08.730734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.730720 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:08:08.788088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.788059 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.788272 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.788093 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.788272 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.788121 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8sj\" (UniqueName: \"kubernetes.io/projected/84eb97d5-3053-4003-98b2-7abaeee1fb64-kube-api-access-4l8sj\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.788783 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.788758 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.790649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.790624 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.845941 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.845911 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8sj\" (UniqueName: \"kubernetes.io/projected/84eb97d5-3053-4003-98b2-7abaeee1fb64-kube-api-access-4l8sj\") pod \"maas-api-revoke-sub-295c6b81-xlwb8\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:08.940255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:08.940176 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:09.312504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:09.312177 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8"] Oct 31 16:08:09.312778 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:08:09.312752 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84eb97d5_3053_4003_98b2_7abaeee1fb64.slice/crio-cface6726cf8d227febe62ef0d2bedd048eac2d1b132e4f65ddf692b8d1f05ba WatchSource:0}: Error finding container cface6726cf8d227febe62ef0d2bedd048eac2d1b132e4f65ddf692b8d1f05ba: Status 404 returned error can't find the container with id cface6726cf8d227febe62ef0d2bedd048eac2d1b132e4f65ddf692b8d1f05ba Oct 31 16:08:09.405505 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:09.405444 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:10.055869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:10.055831 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" event={"ID":"84eb97d5-3053-4003-98b2-7abaeee1fb64","Type":"ContainerStarted","Data":"ec1fc573e87c296504b04a89dcef8b2589025783037bab4f7937061cbf6583e1"} Oct 31 16:08:10.056300 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:10.055883 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" event={"ID":"84eb97d5-3053-4003-98b2-7abaeee1fb64","Type":"ContainerStarted","Data":"cface6726cf8d227febe62ef0d2bedd048eac2d1b132e4f65ddf692b8d1f05ba"} Oct 31 16:08:10.173812 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:10.173764 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" podStartSLOduration=2.173749209 podStartE2EDuration="2.173749209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:10.172338256 +0000 UTC m=+4935.791889666" watchObservedRunningTime="2026-10-31 16:08:10.173749209 +0000 UTC m=+4935.793300620" Oct 31 16:08:11.062896 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:11.062863 4865 generic.go:356] "Generic (PLEG): container finished" podID="84eb97d5-3053-4003-98b2-7abaeee1fb64" containerID="ec1fc573e87c296504b04a89dcef8b2589025783037bab4f7937061cbf6583e1" exitCode=0 Oct 31 16:08:11.063297 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:11.062959 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" event={"ID":"84eb97d5-3053-4003-98b2-7abaeee1fb64","Type":"ContainerDied","Data":"ec1fc573e87c296504b04a89dcef8b2589025783037bab4f7937061cbf6583e1"} Oct 31 16:08:11.407063 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:11.406970 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:11.528728 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:11.528688 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:12.206613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.206588 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:12.322820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.322713 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-openshift-ca\") pod \"84eb97d5-3053-4003-98b2-7abaeee1fb64\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " Oct 31 16:08:12.322820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.322771 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84eb97d5-3053-4003-98b2-7abaeee1fb64-kube-api-access-4l8sj\" (UniqueName: \"kubernetes.io/projected/84eb97d5-3053-4003-98b2-7abaeee1fb64-kube-api-access-4l8sj\") pod \"84eb97d5-3053-4003-98b2-7abaeee1fb64\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " Oct 31 16:08:12.322820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.322800 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-xks-ca\") pod \"84eb97d5-3053-4003-98b2-7abaeee1fb64\" (UID: \"84eb97d5-3053-4003-98b2-7abaeee1fb64\") " Oct 31 16:08:12.323901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.323870 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-openshift-ca" pod "84eb97d5-3053-4003-98b2-7abaeee1fb64" (UID: "84eb97d5-3053-4003-98b2-7abaeee1fb64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:08:12.326286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.326247 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84eb97d5-3053-4003-98b2-7abaeee1fb64-kube-api-access-4l8sj" pod "84eb97d5-3053-4003-98b2-7abaeee1fb64" (UID: "84eb97d5-3053-4003-98b2-7abaeee1fb64"). InnerVolumeSpecName "kube-api-access-4l8sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:08:12.326894 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.326868 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-xks-ca" pod "84eb97d5-3053-4003-98b2-7abaeee1fb64" (UID: "84eb97d5-3053-4003-98b2-7abaeee1fb64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:08:12.424288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.424253 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:12.424288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.424283 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8sj\" (UniqueName: \"kubernetes.io/projected/84eb97d5-3053-4003-98b2-7abaeee1fb64-kube-api-access-4l8sj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:12.424288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:12.424293 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84eb97d5-3053-4003-98b2-7abaeee1fb64-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:13.074759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:13.074731 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" Oct 31 16:08:13.074960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:13.074761 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8" event={"ID":"84eb97d5-3053-4003-98b2-7abaeee1fb64","Type":"ContainerDied","Data":"cface6726cf8d227febe62ef0d2bedd048eac2d1b132e4f65ddf692b8d1f05ba"} Oct 31 16:08:13.074960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:13.074810 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cface6726cf8d227febe62ef0d2bedd048eac2d1b132e4f65ddf692b8d1f05ba" Oct 31 16:08:13.144211 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:13.144176 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:14.102192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:14.102159 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:14.658039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:14.658007 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:17.489200 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:08:17.489161 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:08:18.805114 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:18.805067 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:20.091009 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:20.090974 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:20.556248 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:20.556211 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk"] Oct 31 16:08:20.559614 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:20.559581 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:24.476967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:24.476940 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:26.733820 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:26.733787 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:08:29.868950 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:29.868911 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:34.243108 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:34.243077 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:34.698338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:34.698258 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:34.863891 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:34.863850 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:35.576842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.576808 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" podUID="c4dcaf6b-ed2d-4c1b-afae-b4f195666170" containerName="main" containerID="cri-o://dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2" gracePeriod=45 Oct 31 16:08:35.848073 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.848048 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:08:35.943829 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.943802 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-model-cache\") pod \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " Oct 31 16:08:35.943982 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.943859 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tls-certs\") pod \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " Oct 31 16:08:35.943982 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.943897 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tmp-dir\") pod \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " Oct 31 16:08:35.944064 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944035 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-home\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-home\") pod \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " Oct 31 16:08:35.944064 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944049 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-model-cache" pod "c4dcaf6b-ed2d-4c1b-afae-b4f195666170" (UID: "c4dcaf6b-ed2d-4c1b-afae-b4f195666170"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:08:35.944142 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944081 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-kube-api-access-hjvln\" (UniqueName: \"kubernetes.io/projected/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-kube-api-access-hjvln\") pod \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " Oct 31 16:08:35.944142 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944106 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-dshm\") pod \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\" (UID: \"c4dcaf6b-ed2d-4c1b-afae-b4f195666170\") " Oct 31 16:08:35.944256 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944156 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tmp-dir" pod "c4dcaf6b-ed2d-4c1b-afae-b4f195666170" (UID: "c4dcaf6b-ed2d-4c1b-afae-b4f195666170"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:08:35.944324 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944264 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-home" pod "c4dcaf6b-ed2d-4c1b-afae-b4f195666170" (UID: "c4dcaf6b-ed2d-4c1b-afae-b4f195666170"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:08:35.944396 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944379 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:35.944458 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944400 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:35.944458 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.944412 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:35.946321 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.946300 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-dshm" pod "c4dcaf6b-ed2d-4c1b-afae-b4f195666170" (UID: "c4dcaf6b-ed2d-4c1b-afae-b4f195666170"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:08:35.946321 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.946304 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-kube-api-access-hjvln" pod "c4dcaf6b-ed2d-4c1b-afae-b4f195666170" (UID: "c4dcaf6b-ed2d-4c1b-afae-b4f195666170"). InnerVolumeSpecName "kube-api-access-hjvln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:08:35.946441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:35.946363 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tls-certs" pod "c4dcaf6b-ed2d-4c1b-afae-b4f195666170" (UID: "c4dcaf6b-ed2d-4c1b-afae-b4f195666170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:08:36.045268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.045239 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjvln\" (UniqueName: \"kubernetes.io/projected/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-kube-api-access-hjvln\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:36.045268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.045264 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:36.045268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.045273 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4dcaf6b-ed2d-4c1b-afae-b4f195666170-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:08:36.209144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.209109 4865 generic.go:356] "Generic (PLEG): container finished" podID="c4dcaf6b-ed2d-4c1b-afae-b4f195666170" containerID="dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2" exitCode=0 Oct 31 16:08:36.209312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.209155 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" Oct 31 16:08:36.209312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.209189 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" event={"ID":"c4dcaf6b-ed2d-4c1b-afae-b4f195666170","Type":"ContainerDied","Data":"dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2"} Oct 31 16:08:36.209312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.209236 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk" event={"ID":"c4dcaf6b-ed2d-4c1b-afae-b4f195666170","Type":"ContainerDied","Data":"431a2ed855bb25497068b8163179bba0e19ef82119ca8bfc05ba7217d1e4e4b1"} Oct 31 16:08:36.209312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.209253 4865 scope.go:122] "RemoveContainer" containerID="dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2" Oct 31 16:08:36.219065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.219044 4865 scope.go:122] "RemoveContainer" containerID="dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2" Oct 31 16:08:36.219342 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:08:36.219323 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2\": container with ID starting with dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2 not found: ID does not exist" containerID="dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2" Oct 31 16:08:36.219400 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.219350 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2"} err="failed to get container status \"dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2\": rpc error: code = NotFound desc = could not find container \"dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2\": container with ID starting with dd0d1f3738d309d179ddfc9815e8f4425eaa13da9640ceb7e2f165ea5bec2aa2 not found: ID does not exist" Oct 31 16:08:36.380715 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.380682 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk"] Oct 31 16:08:36.435340 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.435308 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-31dff8/sub-test-model-31dff8-kserve-5c578496d5-l5crk"] Oct 31 16:08:36.495493 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:36.495404 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4dcaf6b-ed2d-4c1b-afae-b4f195666170" path="/var/lib/kubelet/pods/c4dcaf6b-ed2d-4c1b-afae-b4f195666170/volumes" Oct 31 16:08:38.898736 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.898702 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr"] Oct 31 16:08:38.899264 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.899248 4865 cpu_manager.go:414] "containerMap: removing container" podUID="84eb97d5-3053-4003-98b2-7abaeee1fb64" containerName="revoke-keys" Oct 31 16:08:38.899347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.899269 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84eb97d5-3053-4003-98b2-7abaeee1fb64" containerName="revoke-keys" Oct 31 16:08:38.899347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.899288 4865 cpu_manager.go:414] "containerMap: removing container" podUID="c4dcaf6b-ed2d-4c1b-afae-b4f195666170" containerName="main" Oct 31 16:08:38.899347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.899295 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dcaf6b-ed2d-4c1b-afae-b4f195666170" containerName="main" Oct 31 16:08:38.899570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.899441 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="84eb97d5-3053-4003-98b2-7abaeee1fb64" containerName="revoke-keys" Oct 31 16:08:38.899570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.899457 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dcaf6b-ed2d-4c1b-afae-b4f195666170" containerName="main" Oct 31 16:08:38.902709 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.902687 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974387 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-envoy\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974418 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9snj\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-kube-api-access-f9snj\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974440 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-credential-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974517 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istiod-ca-cert\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974644 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-data\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974705 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-token\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974740 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-podinfo\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.974771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974762 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:38.975017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:38.974832 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-certs\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.010807 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.010777 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-140c48-openshift-default-dockercfg-gqjnt\"" Oct 31 16:08:39.066083 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.066049 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr"] Oct 31 16:08:39.075455 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075429 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-token\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075461 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-podinfo\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075524 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075561 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-certs\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075789 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075616 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-envoy\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075789 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075639 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9snj\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-kube-api-access-f9snj\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075789 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075671 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-credential-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075789 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075708 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istiod-ca-cert\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.075789 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.075775 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-data\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.076079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.076028 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-credential-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.076079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.076038 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.076164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.076107 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-data\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.076409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.076392 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istiod-ca-cert\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.076456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.076311 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-certs\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.078143 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.078123 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-podinfo\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.078456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.078436 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-envoy\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.159744 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.159684 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-token\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.172015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.171989 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9snj\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-kube-api-access-f9snj\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.173163 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.173144 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:08:39.215336 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.215300 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:39.639561 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.639534 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr"] Oct 31 16:08:39.641090 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:08:39.641065 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9b07b3_ca86_4de9_bf2c_0a1e7e4cde23.slice/crio-a6905ca1bd95be39dee7360bd7baf54f3e79357e103f444aa39f5631ea428d1d WatchSource:0}: Error finding container a6905ca1bd95be39dee7360bd7baf54f3e79357e103f444aa39f5631ea428d1d: Status 404 returned error can't find the container with id a6905ca1bd95be39dee7360bd7baf54f3e79357e103f444aa39f5631ea428d1d Oct 31 16:08:39.644214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.644160 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:08:39.644338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.644265 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:08:39.644338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.644301 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:08:39.761636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:39.761605 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:40.237050 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:40.237009 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" event={"ID":"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23","Type":"ContainerStarted","Data":"505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230"} Oct 31 16:08:40.237555 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:40.237068 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" event={"ID":"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23","Type":"ContainerStarted","Data":"a6905ca1bd95be39dee7360bd7baf54f3e79357e103f444aa39f5631ea428d1d"} Oct 31 16:08:40.469256 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:40.469201 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" podStartSLOduration=2.469186456 podStartE2EDuration="2.469186456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:40.415954393 +0000 UTC m=+4966.035505807" watchObservedRunningTime="2026-10-31 16:08:40.469186456 +0000 UTC m=+4966.088737940" Oct 31 16:08:40.519051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:40.518946 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:41.217372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:41.217339 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:41.217756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:41.217732 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" start-of-body= Oct 31 16:08:41.217962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:41.217936 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" Oct 31 16:08:42.218902 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:42.218860 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" start-of-body= Oct 31 16:08:42.219293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:42.218926 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" Oct 31 16:08:43.218529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:43.217824 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" start-of-body= Oct 31 16:08:43.218529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:43.217886 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" Oct 31 16:08:43.275125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:43.275092 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:43.419933 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:43.419894 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:44.218854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:44.218814 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" start-of-body= Oct 31 16:08:44.219076 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:44.218877 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" Oct 31 16:08:44.418528 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:44.416999 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:45.215878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:45.215837 4865 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" start-of-body= Oct 31 16:08:45.216191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:45.216161 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.137:15021/healthz/ready\": dial tcp 10.132.1.137:15021: connect: connection refused" Oct 31 16:08:45.931066 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:45.931036 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:46.218186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:46.218155 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:46.219753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:46.219732 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:46.220158 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:46.220137 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:46.220905 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:46.220890 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:08:47.489449 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:08:47.489412 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:08:52.123527 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:52.123491 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:08:52.213661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:52.213626 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:56.019719 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:08:56.019685 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:00.196006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:00.195973 4865 scope.go:122] "RemoveContainer" containerID="8fb146b2d052bbba45f5b3c0dc5964d888c342d23cf219996ed7864635aa8716" Oct 31 16:09:00.205332 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:00.205306 4865 scope.go:122] "RemoveContainer" containerID="afd407f9214e3d99777ab538f1d25e8c22d877d927bbe140c691549e1990600f" Oct 31 16:09:02.008349 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:02.008312 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:03.045392 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:03.045356 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w"] Oct 31 16:09:03.064575 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:03.064545 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae4b3099-xxf7w"] Oct 31 16:09:04.502422 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:04.502381 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fea8f1-a4d4-4b15-ae66-77fcf02ec623" path="/var/lib/kubelet/pods/53fea8f1-a4d4-4b15-ae66-77fcf02ec623/volumes" Oct 31 16:09:06.920764 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:06.920683 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:08.382184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:08.382150 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:08.574176 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:08.574119 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:09.922617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:09.922556 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:10.511797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:10.511767 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf"] Oct 31 16:09:10.512036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:10.512008 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" containerID="cri-o://d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce" gracePeriod=30 Oct 31 16:09:10.515825 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:10.515798 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:11.257938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:11.257880 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:12.009442 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:12.009386 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:13.582999 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:13.582929 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:14.045461 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:14.045431 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt"] Oct 31 16:09:14.098116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:14.098088 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0410ee3-mkhgt"] Oct 31 16:09:14.495004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:14.494972 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49ffa24-0879-4b9a-b393-59df15606b55" path="/var/lib/kubelet/pods/a49ffa24-0879-4b9a-b393-59df15606b55/volumes" Oct 31 16:09:15.048832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.048750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:15.761409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.761386 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 16:09:15.845345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845281 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istiod-ca-cert\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845355 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-kube-api-access-g5hfd\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-kube-api-access-g5hfd\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845383 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-credential-socket\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845432 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-podinfo\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845481 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-socket\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845538 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-envoy\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845593 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-certs\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845621 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-token\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-token\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845671 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-data\") pod \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\" (UID: \"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21\") " Oct 31 16:09:15.845732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845710 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-credential-socket" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.845732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845719 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istiod-ca-cert" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:15.846094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845870 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-socket" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.846094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.845891 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-certs" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.846094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.846038 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.846094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.846061 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.846094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.846071 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.846094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.846075 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-data" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.846094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.846084 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.847874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.847849 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-kube-api-access-g5hfd" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "kube-api-access-g5hfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:15.847874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.847851 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-podinfo" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:09:15.848010 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.847849 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-envoy" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.848095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.848070 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-token" pod "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" (UID: "7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:15.946970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.946939 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.946970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.946968 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.947170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.946983 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hfd\" (UniqueName: \"kubernetes.io/projected/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-kube-api-access-g5hfd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.947170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.946995 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.947170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:15.947008 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.436804 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.436763 4865 generic.go:356] "Generic (PLEG): container finished" podID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerID="d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce" exitCode=0 Oct 31 16:09:16.437318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.436817 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" Oct 31 16:09:16.437318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.436840 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" event={"ID":"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21","Type":"ContainerDied","Data":"d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce"} Oct 31 16:09:16.437318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.436909 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf" event={"ID":"7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21","Type":"ContainerDied","Data":"821bae8a431d3f735bbe70577a820b8f37624d4a4639093377d46aeec7302eb8"} Oct 31 16:09:16.437318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.436940 4865 scope.go:122] "RemoveContainer" containerID="d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce" Oct 31 16:09:16.447005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.446977 4865 scope.go:122] "RemoveContainer" containerID="cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97" Oct 31 16:09:16.456179 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.456154 4865 scope.go:122] "RemoveContainer" containerID="d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce" Oct 31 16:09:16.456449 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:16.456426 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce\": container with ID starting with d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce not found: ID does not exist" containerID="d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce" Oct 31 16:09:16.456530 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.456459 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce"} err="failed to get container status \"d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce\": rpc error: code = NotFound desc = could not find container \"d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce\": container with ID starting with d5b65dba67c626112bf52aa31086e6cf1823b7067bb9de6c901e360d19fc16ce not found: ID does not exist" Oct 31 16:09:16.456530 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.456501 4865 scope.go:122] "RemoveContainer" containerID="cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97" Oct 31 16:09:16.456738 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:16.456723 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97\": container with ID starting with cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97 not found: ID does not exist" containerID="cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97" Oct 31 16:09:16.456780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.456744 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97"} err="failed to get container status \"cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97\": rpc error: code = NotFound desc = could not find container \"cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97\": container with ID starting with cd67c150d6bd64b7c4214206fa5c0691f943648c9ba59f23ebfe0fe3d0706f97 not found: ID does not exist" Oct 31 16:09:16.661608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.661574 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf"] Oct 31 16:09:16.714684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.714655 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-8d8pf"] Oct 31 16:09:16.719672 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.719645 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl"] Oct 31 16:09:16.911708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:16.911671 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:17.038254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:17.038170 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:17.163402 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:17.163356 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:17.286045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:17.286016 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:17.488554 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:17.488519 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:09:18.495603 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:18.495563 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" path="/var/lib/kubelet/pods/7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21/volumes" Oct 31 16:09:21.469256 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:21.469207 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:21.720327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:21.720238 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" podUID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" containerName="payload-pre-processing" containerID="cri-o://e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013" gracePeriod=25 Oct 31 16:09:22.936200 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:22.936136 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:23.063450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:23.063404 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:23.294175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:23.294142 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:24.468023 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:24.467990 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:24.950293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:24.950189 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:24.950695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:24.950675 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr"] Oct 31 16:09:24.950907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:24.950887 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" containerID="cri-o://505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230" gracePeriod=30 Oct 31 16:09:25.398296 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:25.398257 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:26.003552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.003516 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4"] Oct 31 16:09:26.003952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.003897 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" Oct 31 16:09:26.003952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.003908 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" Oct 31 16:09:26.004035 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.003990 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" Oct 31 16:09:26.008498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.008452 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.041345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.041322 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.041489 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.041360 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.041547 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.041490 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82m4\" (UniqueName: \"kubernetes.io/projected/ccd82be7-4054-4dd7-95f1-d6d460cd359f-kube-api-access-c82m4\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.064994 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.064966 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4"] Oct 31 16:09:26.113159 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.113123 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:09:26.113376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.113234 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:09:26.113376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.113245 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:09:26.113376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.113258 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:09:26.142807 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.142783 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82m4\" (UniqueName: \"kubernetes.io/projected/ccd82be7-4054-4dd7-95f1-d6d460cd359f-kube-api-access-c82m4\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.142938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.142831 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.143025 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.142957 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.143566 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.143550 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.145237 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.145215 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.202420 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.202392 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82m4\" (UniqueName: \"kubernetes.io/projected/ccd82be7-4054-4dd7-95f1-d6d460cd359f-kube-api-access-c82m4\") pod \"maas-api-revoke-sub-51680b8c-lftk4\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.320941 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.320880 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:26.689106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:26.689077 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4"] Oct 31 16:09:26.689937 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:09:26.689907 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd82be7_4054_4dd7_95f1_d6d460cd359f.slice/crio-e964b06207a611aae5b3c62c427f2d9740fd80214e25f5d3f4a8571da01e7cae WatchSource:0}: Error finding container e964b06207a611aae5b3c62c427f2d9740fd80214e25f5d3f4a8571da01e7cae: Status 404 returned error can't find the container with id e964b06207a611aae5b3c62c427f2d9740fd80214e25f5d3f4a8571da01e7cae Oct 31 16:09:27.502107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:27.502060 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" event={"ID":"ccd82be7-4054-4dd7-95f1-d6d460cd359f","Type":"ContainerStarted","Data":"7c13f323e2ef268fd72c9343f215b15d78465e3ea79f9702883d380e29f9ae66"} Oct 31 16:09:27.502624 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:27.502128 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" event={"ID":"ccd82be7-4054-4dd7-95f1-d6d460cd359f","Type":"ContainerStarted","Data":"e964b06207a611aae5b3c62c427f2d9740fd80214e25f5d3f4a8571da01e7cae"} Oct 31 16:09:27.617109 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:27.617061 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" podStartSLOduration=2.617048001 podStartE2EDuration="2.617048001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:27.616291126 +0000 UTC m=+5013.235842532" watchObservedRunningTime="2026-10-31 16:09:27.617048001 +0000 UTC m=+5013.236599411" Oct 31 16:09:28.329784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:28.329750 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:28.508623 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:28.508592 4865 generic.go:356] "Generic (PLEG): container finished" podID="ccd82be7-4054-4dd7-95f1-d6d460cd359f" containerID="7c13f323e2ef268fd72c9343f215b15d78465e3ea79f9702883d380e29f9ae66" exitCode=0 Oct 31 16:09:28.509036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:28.508652 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" event={"ID":"ccd82be7-4054-4dd7-95f1-d6d460cd359f","Type":"ContainerDied","Data":"7c13f323e2ef268fd72c9343f215b15d78465e3ea79f9702883d380e29f9ae66"} Oct 31 16:09:29.040302 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.040263 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:09:29.656436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.656408 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:29.675975 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.675944 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-xks-ca\") pod \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " Oct 31 16:09:29.676164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.676014 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-openshift-ca\") pod \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " Oct 31 16:09:29.676164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.676055 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd82be7-4054-4dd7-95f1-d6d460cd359f-kube-api-access-c82m4\" (UniqueName: \"kubernetes.io/projected/ccd82be7-4054-4dd7-95f1-d6d460cd359f-kube-api-access-c82m4\") pod \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\" (UID: \"ccd82be7-4054-4dd7-95f1-d6d460cd359f\") " Oct 31 16:09:29.676444 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.676414 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-openshift-ca" pod "ccd82be7-4054-4dd7-95f1-d6d460cd359f" (UID: "ccd82be7-4054-4dd7-95f1-d6d460cd359f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:29.678322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.678293 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-xks-ca" pod "ccd82be7-4054-4dd7-95f1-d6d460cd359f" (UID: "ccd82be7-4054-4dd7-95f1-d6d460cd359f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:09:29.678439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.678380 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd82be7-4054-4dd7-95f1-d6d460cd359f-kube-api-access-c82m4" pod "ccd82be7-4054-4dd7-95f1-d6d460cd359f" (UID: "ccd82be7-4054-4dd7-95f1-d6d460cd359f"). InnerVolumeSpecName "kube-api-access-c82m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:29.781539 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.781435 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:29.781539 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.781495 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd82be7-4054-4dd7-95f1-d6d460cd359f-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:29.781539 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:29.781512 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82m4\" (UniqueName: \"kubernetes.io/projected/ccd82be7-4054-4dd7-95f1-d6d460cd359f-kube-api-access-c82m4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.081407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.081368 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" podUID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:09:30.189697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.189671 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:09:30.287160 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287128 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-data\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287379 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287184 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-envoy\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287379 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287216 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-certs\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287379 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287250 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-credential-socket\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287379 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287278 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-socket\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287381 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-token\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-token\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287443 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istiod-ca-cert\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287449 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-data" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:30.287660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287591 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-certs" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:30.287660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287606 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-kube-api-access-f9snj\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-kube-api-access-f9snj\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287611 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-credential-socket" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:30.287660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287652 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-podinfo\") pod \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\" (UID: \"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23\") " Oct 31 16:09:30.287995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287742 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-socket" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:30.287995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287823 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istiod-ca-cert" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:30.287995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.287989 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.288170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.288007 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.288170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.288022 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.288170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.288037 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.288170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.288049 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.289951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.289914 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-podinfo" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:09:30.289951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.289934 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-token" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:30.290161 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.290144 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-envoy" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:30.290223 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.290205 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-kube-api-access-f9snj" pod "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" (UID: "fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23"). InnerVolumeSpecName "kube-api-access-f9snj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:30.389261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.389177 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.389261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.389208 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.389261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.389220 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9snj\" (UniqueName: \"kubernetes.io/projected/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-kube-api-access-f9snj\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.389261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.389236 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:30.414395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.414344 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m"] Oct 31 16:09:30.416644 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.416619 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x"] Oct 31 16:09:30.520795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.520766 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" Oct 31 16:09:30.520954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.520789 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4" event={"ID":"ccd82be7-4054-4dd7-95f1-d6d460cd359f","Type":"ContainerDied","Data":"e964b06207a611aae5b3c62c427f2d9740fd80214e25f5d3f4a8571da01e7cae"} Oct 31 16:09:30.520954 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.520840 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e964b06207a611aae5b3c62c427f2d9740fd80214e25f5d3f4a8571da01e7cae" Oct 31 16:09:30.522290 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.522267 4865 generic.go:356] "Generic (PLEG): container finished" podID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerID="505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230" exitCode=0 Oct 31 16:09:30.522430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.522316 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" Oct 31 16:09:30.522430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.522324 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" event={"ID":"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23","Type":"ContainerDied","Data":"505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230"} Oct 31 16:09:30.522430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.522358 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr" event={"ID":"fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23","Type":"ContainerDied","Data":"a6905ca1bd95be39dee7360bd7baf54f3e79357e103f444aa39f5631ea428d1d"} Oct 31 16:09:30.522430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.522380 4865 scope.go:122] "RemoveContainer" containerID="505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230" Oct 31 16:09:30.531817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.531801 4865 scope.go:122] "RemoveContainer" containerID="505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230" Oct 31 16:09:30.532140 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:30.532118 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230\": container with ID starting with 505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230 not found: ID does not exist" containerID="505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230" Oct 31 16:09:30.532205 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.532152 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230"} err="failed to get container status \"505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230\": rpc error: code = NotFound desc = could not find container \"505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230\": container with ID starting with 505857863f6fc569394b3ce38a14f1eb24ddbec6daf2d285267e7288bcb79230 not found: ID does not exist" Oct 31 16:09:30.534000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.533979 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:30.646049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.646024 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr"] Oct 31 16:09:30.696713 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:30.696681 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-wcjbr"] Oct 31 16:09:32.471898 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:32.471859 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:32.494802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:32.494770 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" path="/var/lib/kubelet/pods/fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23/volumes" Oct 31 16:09:33.171884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:33.171842 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:37.839266 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:37.839231 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:37.945116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:37.945077 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5"] Oct 31 16:09:38.051698 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.051661 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj"] Oct 31 16:09:38.269943 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.269915 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w"] Oct 31 16:09:38.270315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270302 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ccd82be7-4054-4dd7-95f1-d6d460cd359f" containerName="revoke-keys" Oct 31 16:09:38.270378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270317 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd82be7-4054-4dd7-95f1-d6d460cd359f" containerName="revoke-keys" Oct 31 16:09:38.270378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270339 4865 cpu_manager.go:414] "containerMap: removing container" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" Oct 31 16:09:38.270378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270344 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" Oct 31 16:09:38.270378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270367 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" Oct 31 16:09:38.270378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270372 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" Oct 31 16:09:38.270583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270428 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de2ffbe-06bd-4a52-8bf2-b583cd1a8a21" containerName="istio-proxy" Oct 31 16:09:38.270583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270441 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9b07b3-ca86-4de9-bf2c-0a1e7e4cde23" containerName="istio-proxy" Oct 31 16:09:38.270583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.270449 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd82be7-4054-4dd7-95f1-d6d460cd359f" containerName="revoke-keys" Oct 31 16:09:38.273346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.273328 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.293800 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.293770 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:38.327438 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.327413 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:09:38.327676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.327498 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:09:38.327826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.327689 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:09:38.329664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.329643 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w"] Oct 31 16:09:38.362451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.362413 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.362657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.362499 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhhf\" (UniqueName: \"kubernetes.io/projected/6ebd0091-6352-419d-a292-0fac0c76d495-kube-api-access-wlhhf\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.362657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.362623 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.377983 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.377949 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:09:38.417134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.417093 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:38.463887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.463855 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.464073 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.463894 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhhf\" (UniqueName: \"kubernetes.io/projected/6ebd0091-6352-419d-a292-0fac0c76d495-kube-api-access-wlhhf\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.464073 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.463936 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.464536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.464499 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.466531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.466507 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.521728 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.521651 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhhf\" (UniqueName: \"kubernetes.io/projected/6ebd0091-6352-419d-a292-0fac0c76d495-kube-api-access-wlhhf\") pod \"maas-api-revoke-sub-0f0b964e-q5f2w\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.585732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.585699 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:38.742218 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.742155 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:38.964219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:38.964188 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w"] Oct 31 16:09:38.965316 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:09:38.965286 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebd0091_6352_419d_a292_0fac0c76d495.slice/crio-083c9b9bf1331258adffafdf73f6591b0d7c4bb828c4b4b7e3d3656dbbf84d29 WatchSource:0}: Error finding container 083c9b9bf1331258adffafdf73f6591b0d7c4bb828c4b4b7e3d3656dbbf84d29: Status 404 returned error can't find the container with id 083c9b9bf1331258adffafdf73f6591b0d7c4bb828c4b4b7e3d3656dbbf84d29 Oct 31 16:09:39.572109 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:39.572066 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" event={"ID":"6ebd0091-6352-419d-a292-0fac0c76d495","Type":"ContainerStarted","Data":"f83e6add16a1c651518c49925a53c585afde5ce36dd73c2856b9498cdd6a4e2c"} Oct 31 16:09:39.572566 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:39.572133 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" event={"ID":"6ebd0091-6352-419d-a292-0fac0c76d495","Type":"ContainerStarted","Data":"083c9b9bf1331258adffafdf73f6591b0d7c4bb828c4b4b7e3d3656dbbf84d29"} Oct 31 16:09:39.688343 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:39.688287 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" podStartSLOduration=1.6882727960000001 podStartE2EDuration="1.688272796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:39.686209533 +0000 UTC m=+5025.305760948" watchObservedRunningTime="2026-10-31 16:09:39.688272796 +0000 UTC m=+5025.307824209" Oct 31 16:09:40.080861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:40.080826 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" podUID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:09:40.578582 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:40.578544 4865 generic.go:356] "Generic (PLEG): container finished" podID="6ebd0091-6352-419d-a292-0fac0c76d495" containerID="f83e6add16a1c651518c49925a53c585afde5ce36dd73c2856b9498cdd6a4e2c" exitCode=0 Oct 31 16:09:40.578793 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:40.578630 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" event={"ID":"6ebd0091-6352-419d-a292-0fac0c76d495","Type":"ContainerDied","Data":"f83e6add16a1c651518c49925a53c585afde5ce36dd73c2856b9498cdd6a4e2c"} Oct 31 16:09:41.712283 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.712258 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:41.792346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.792308 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebd0091-6352-419d-a292-0fac0c76d495-kube-api-access-wlhhf\" (UniqueName: \"kubernetes.io/projected/6ebd0091-6352-419d-a292-0fac0c76d495-kube-api-access-wlhhf\") pod \"6ebd0091-6352-419d-a292-0fac0c76d495\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " Oct 31 16:09:41.792592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.792368 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-openshift-ca\") pod \"6ebd0091-6352-419d-a292-0fac0c76d495\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " Oct 31 16:09:41.792592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.792415 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-xks-ca\") pod \"6ebd0091-6352-419d-a292-0fac0c76d495\" (UID: \"6ebd0091-6352-419d-a292-0fac0c76d495\") " Oct 31 16:09:41.792826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.792798 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-openshift-ca" pod "6ebd0091-6352-419d-a292-0fac0c76d495" (UID: "6ebd0091-6352-419d-a292-0fac0c76d495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:41.794765 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.794734 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-xks-ca" pod "6ebd0091-6352-419d-a292-0fac0c76d495" (UID: "6ebd0091-6352-419d-a292-0fac0c76d495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:09:41.794863 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.794761 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebd0091-6352-419d-a292-0fac0c76d495-kube-api-access-wlhhf" pod "6ebd0091-6352-419d-a292-0fac0c76d495" (UID: "6ebd0091-6352-419d-a292-0fac0c76d495"). InnerVolumeSpecName "kube-api-access-wlhhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:41.894253 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.894218 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:41.894407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.894255 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhhf\" (UniqueName: \"kubernetes.io/projected/6ebd0091-6352-419d-a292-0fac0c76d495-kube-api-access-wlhhf\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:41.894407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.894266 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd0091-6352-419d-a292-0fac0c76d495-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:41.974393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:41.974366 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 16:09:42.096709 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.096612 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e0e1ff-f21e-4725-b0c6-a523bb319651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e0e1ff-f21e-4725-b0c6-a523bb319651-plugins-config-volume\") pod \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " Oct 31 16:09:42.096709 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.096688 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e0e1ff-f21e-4725-b0c6-a523bb319651-kube-api-access-7gs89\" (UniqueName: \"kubernetes.io/projected/e2e0e1ff-f21e-4725-b0c6-a523bb319651-kube-api-access-7gs89\") pod \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\" (UID: \"e2e0e1ff-f21e-4725-b0c6-a523bb319651\") " Oct 31 16:09:42.097012 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.096988 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e0e1ff-f21e-4725-b0c6-a523bb319651-plugins-config-volume" pod "e2e0e1ff-f21e-4725-b0c6-a523bb319651" (UID: "e2e0e1ff-f21e-4725-b0c6-a523bb319651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:42.097135 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.097117 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e0e1ff-f21e-4725-b0c6-a523bb319651-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:42.098894 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.098868 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e0e1ff-f21e-4725-b0c6-a523bb319651-kube-api-access-7gs89" pod "e2e0e1ff-f21e-4725-b0c6-a523bb319651" (UID: "e2e0e1ff-f21e-4725-b0c6-a523bb319651"). InnerVolumeSpecName "kube-api-access-7gs89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:42.198199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.198165 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gs89\" (UniqueName: \"kubernetes.io/projected/e2e0e1ff-f21e-4725-b0c6-a523bb319651-kube-api-access-7gs89\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:42.511089 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.511061 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:09:42.590919 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.590883 4865 generic.go:356] "Generic (PLEG): container finished" podID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" containerID="e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013" exitCode=0 Oct 31 16:09:42.591105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.590927 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" Oct 31 16:09:42.591105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.590964 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" event={"ID":"e2e0e1ff-f21e-4725-b0c6-a523bb319651","Type":"ContainerDied","Data":"e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013"} Oct 31 16:09:42.591105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.591017 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl" event={"ID":"e2e0e1ff-f21e-4725-b0c6-a523bb319651","Type":"ContainerDied","Data":"5de4605f5015ed624474b8c928a573091e020335eb64b0b3ce64da6b973cc834"} Oct 31 16:09:42.591105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.591039 4865 scope.go:122] "RemoveContainer" containerID="e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013" Oct 31 16:09:42.592635 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.592614 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" event={"ID":"6ebd0091-6352-419d-a292-0fac0c76d495","Type":"ContainerDied","Data":"083c9b9bf1331258adffafdf73f6591b0d7c4bb828c4b4b7e3d3656dbbf84d29"} Oct 31 16:09:42.592908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.592638 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w" Oct 31 16:09:42.592908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.592655 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="083c9b9bf1331258adffafdf73f6591b0d7c4bb828c4b4b7e3d3656dbbf84d29" Oct 31 16:09:42.602562 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.602540 4865 scope.go:122] "RemoveContainer" containerID="e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013" Oct 31 16:09:42.602846 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:42.602824 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013\": container with ID starting with e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013 not found: ID does not exist" containerID="e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013" Oct 31 16:09:42.602947 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.602851 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013"} err="failed to get container status \"e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013\": rpc error: code = NotFound desc = could not find container \"e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013\": container with ID starting with e1928b366b1e389f7839127072a3e1cb1c68b3650141dd3bce51a676efd2a013 not found: ID does not exist" Oct 31 16:09:42.756711 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.756617 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl"] Oct 31 16:09:42.771668 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.771628 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-140c48-c667d4587-sqmtl"] Oct 31 16:09:42.945714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:42.945660 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" podUID="5f0e1dbd-d3a0-4528-900c-529504a0fbed" containerName="payload-processing" containerID="cri-o://ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551" gracePeriod=25 Oct 31 16:09:43.052594 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.052514 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" podUID="4663720c-0379-485b-9af9-7e90dcfc4fc7" containerName="payload-pre-processing" containerID="cri-o://b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c" gracePeriod=25 Oct 31 16:09:43.206845 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.206815 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 16:09:43.304088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.304014 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 16:09:43.307105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.307080 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0e1dbd-d3a0-4528-900c-529504a0fbed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0e1dbd-d3a0-4528-900c-529504a0fbed-plugins-config-volume\") pod \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " Oct 31 16:09:43.307228 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.307115 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0e1dbd-d3a0-4528-900c-529504a0fbed-kube-api-access-crmzv\" (UniqueName: \"kubernetes.io/projected/5f0e1dbd-d3a0-4528-900c-529504a0fbed-kube-api-access-crmzv\") pod \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\" (UID: \"5f0e1dbd-d3a0-4528-900c-529504a0fbed\") " Oct 31 16:09:43.307448 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.307427 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0e1dbd-d3a0-4528-900c-529504a0fbed-plugins-config-volume" pod "5f0e1dbd-d3a0-4528-900c-529504a0fbed" (UID: "5f0e1dbd-d3a0-4528-900c-529504a0fbed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:43.309367 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.309343 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0e1dbd-d3a0-4528-900c-529504a0fbed-kube-api-access-crmzv" pod "5f0e1dbd-d3a0-4528-900c-529504a0fbed" (UID: "5f0e1dbd-d3a0-4528-900c-529504a0fbed"). InnerVolumeSpecName "kube-api-access-crmzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:43.407786 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.407742 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4663720c-0379-485b-9af9-7e90dcfc4fc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663720c-0379-485b-9af9-7e90dcfc4fc7-plugins-config-volume\") pod \"4663720c-0379-485b-9af9-7e90dcfc4fc7\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " Oct 31 16:09:43.408004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.407799 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4663720c-0379-485b-9af9-7e90dcfc4fc7-kube-api-access-ltmdk\" (UniqueName: \"kubernetes.io/projected/4663720c-0379-485b-9af9-7e90dcfc4fc7-kube-api-access-ltmdk\") pod \"4663720c-0379-485b-9af9-7e90dcfc4fc7\" (UID: \"4663720c-0379-485b-9af9-7e90dcfc4fc7\") " Oct 31 16:09:43.408145 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.408125 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f0e1dbd-d3a0-4528-900c-529504a0fbed-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:43.408257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.408151 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmzv\" (UniqueName: \"kubernetes.io/projected/5f0e1dbd-d3a0-4528-900c-529504a0fbed-kube-api-access-crmzv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:43.408257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.408158 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4663720c-0379-485b-9af9-7e90dcfc4fc7-plugins-config-volume" pod "4663720c-0379-485b-9af9-7e90dcfc4fc7" (UID: "4663720c-0379-485b-9af9-7e90dcfc4fc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:43.410198 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.410173 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4663720c-0379-485b-9af9-7e90dcfc4fc7-kube-api-access-ltmdk" pod "4663720c-0379-485b-9af9-7e90dcfc4fc7" (UID: "4663720c-0379-485b-9af9-7e90dcfc4fc7"). InnerVolumeSpecName "kube-api-access-ltmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:43.508900 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.508855 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663720c-0379-485b-9af9-7e90dcfc4fc7-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:43.508900 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.508897 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmdk\" (UniqueName: \"kubernetes.io/projected/4663720c-0379-485b-9af9-7e90dcfc4fc7-kube-api-access-ltmdk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:43.598869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.598780 4865 generic.go:356] "Generic (PLEG): container finished" podID="4663720c-0379-485b-9af9-7e90dcfc4fc7" containerID="b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c" exitCode=0 Oct 31 16:09:43.598869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.598829 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" Oct 31 16:09:43.599123 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.598863 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" event={"ID":"4663720c-0379-485b-9af9-7e90dcfc4fc7","Type":"ContainerDied","Data":"b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c"} Oct 31 16:09:43.599123 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.598917 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj" event={"ID":"4663720c-0379-485b-9af9-7e90dcfc4fc7","Type":"ContainerDied","Data":"f09ea04531ab78c8a5f5c9b27fda90f63a6270515145a66264d521333a94dfcc"} Oct 31 16:09:43.599123 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.598947 4865 scope.go:122] "RemoveContainer" containerID="b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c" Oct 31 16:09:43.600617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.600592 4865 generic.go:356] "Generic (PLEG): container finished" podID="5f0e1dbd-d3a0-4528-900c-529504a0fbed" containerID="ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551" exitCode=0 Oct 31 16:09:43.600729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.600645 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" Oct 31 16:09:43.600729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.600677 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" event={"ID":"5f0e1dbd-d3a0-4528-900c-529504a0fbed","Type":"ContainerDied","Data":"ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551"} Oct 31 16:09:43.600729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.600727 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5" event={"ID":"5f0e1dbd-d3a0-4528-900c-529504a0fbed","Type":"ContainerDied","Data":"83d3864b6de69083ea06980555a6edc040a49309f523b1efaeb52b6b39ebdc80"} Oct 31 16:09:43.609654 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.609638 4865 scope.go:122] "RemoveContainer" containerID="b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c" Oct 31 16:09:43.610086 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:43.609957 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c\": container with ID starting with b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c not found: ID does not exist" containerID="b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c" Oct 31 16:09:43.610086 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.609981 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c"} err="failed to get container status \"b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c\": rpc error: code = NotFound desc = could not find container \"b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c\": container with ID starting with b2e4a72bad6f50dca0f24829894fb01c897445947ce7e0848629a7f90feabb8c not found: ID does not exist" Oct 31 16:09:43.610086 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.609998 4865 scope.go:122] "RemoveContainer" containerID="ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551" Oct 31 16:09:43.618790 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.618773 4865 scope.go:122] "RemoveContainer" containerID="ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551" Oct 31 16:09:43.619058 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:43.619037 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551\": container with ID starting with ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551 not found: ID does not exist" containerID="ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551" Oct 31 16:09:43.619111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.619066 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551"} err="failed to get container status \"ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551\": rpc error: code = NotFound desc = could not find container \"ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551\": container with ID starting with ba769626f3ca711f33bff19afcdc8c38bf4164d444062b2047bb408be5f18551 not found: ID does not exist" Oct 31 16:09:43.768439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.768402 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj"] Oct 31 16:09:43.770526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.770492 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-56bc96-79c7d69bdc-fslkj"] Oct 31 16:09:43.880701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.880598 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5"] Oct 31 16:09:43.883874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:43.883849 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-56bc96-6f47b4bffc-mrpb5"] Oct 31 16:09:44.374447 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:44.374409 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:44.494968 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:44.494930 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4663720c-0379-485b-9af9-7e90dcfc4fc7" path="/var/lib/kubelet/pods/4663720c-0379-485b-9af9-7e90dcfc4fc7/volumes" Oct 31 16:09:44.495405 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:44.495384 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0e1dbd-d3a0-4528-900c-529504a0fbed" path="/var/lib/kubelet/pods/5f0e1dbd-d3a0-4528-900c-529504a0fbed/volumes" Oct 31 16:09:44.495811 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:44.495793 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" path="/var/lib/kubelet/pods/e2e0e1ff-f21e-4725-b0c6-a523bb319651/volumes" Oct 31 16:09:45.435624 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.435588 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" containerID="cri-o://dc8cb268d8f785345c29df9afe6ad77ba43ccf6ce0ea62373175399bb269e656" gracePeriod=45 Oct 31 16:09:45.436067 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.435935 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" containerID="cri-o://7bd6756788eee17b8eddcebc6a90a74d77b2ed47e67f7f56f6b45c835a9fbbe1" gracePeriod=45 Oct 31 16:09:45.594667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.594628 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Oct 31 16:09:45.617981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.617949 4865 generic.go:356] "Generic (PLEG): container finished" podID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerID="dc8cb268d8f785345c29df9afe6ad77ba43ccf6ce0ea62373175399bb269e656" exitCode=0 Oct 31 16:09:45.618153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.618019 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" event={"ID":"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66","Type":"ContainerDied","Data":"dc8cb268d8f785345c29df9afe6ad77ba43ccf6ce0ea62373175399bb269e656"} Oct 31 16:09:45.619986 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.619955 4865 generic.go:356] "Generic (PLEG): container finished" podID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerID="7bd6756788eee17b8eddcebc6a90a74d77b2ed47e67f7f56f6b45c835a9fbbe1" exitCode=0 Oct 31 16:09:45.620132 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.619999 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" event={"ID":"352ed02e-2cd6-4d2c-b978-a55077005e59","Type":"ContainerDied","Data":"7bd6756788eee17b8eddcebc6a90a74d77b2ed47e67f7f56f6b45c835a9fbbe1"} Oct 31 16:09:45.630458 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.630420 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": dial tcp 10.132.0.115:8000: connect: connection refused" Oct 31 16:09:45.694933 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.694861 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 16:09:45.720334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.720311 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 16:09:45.831951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.831919 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-model-cache\") pod \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " Oct 31 16:09:45.831951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.831956 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tmp-dir\") pod \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " Oct 31 16:09:45.832242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.831994 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-kube-api-access-hmfv8\" (UniqueName: \"kubernetes.io/projected/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-kube-api-access-hmfv8\") pod \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " Oct 31 16:09:45.832242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832026 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352ed02e-2cd6-4d2c-b978-a55077005e59-kube-api-access-7qpb7\" (UniqueName: \"kubernetes.io/projected/352ed02e-2cd6-4d2c-b978-a55077005e59-kube-api-access-7qpb7\") pod \"352ed02e-2cd6-4d2c-b978-a55077005e59\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " Oct 31 16:09:45.832242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832053 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tls-certs\" (UniqueName: \"kubernetes.io/secret/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tls-certs\") pod \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " Oct 31 16:09:45.832242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832082 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-dshm\") pod \"352ed02e-2cd6-4d2c-b978-a55077005e59\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " Oct 31 16:09:45.832242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832129 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-home\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-home\") pod \"352ed02e-2cd6-4d2c-b978-a55077005e59\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " Oct 31 16:09:45.832242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832209 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-dshm\") pod \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " Oct 31 16:09:45.832621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832262 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-tmp-dir\") pod \"352ed02e-2cd6-4d2c-b978-a55077005e59\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " Oct 31 16:09:45.832621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832378 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-model-cache\") pod \"352ed02e-2cd6-4d2c-b978-a55077005e59\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " Oct 31 16:09:45.832621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832425 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/352ed02e-2cd6-4d2c-b978-a55077005e59-tls-certs\" (UniqueName: \"kubernetes.io/secret/352ed02e-2cd6-4d2c-b978-a55077005e59-tls-certs\") pod \"352ed02e-2cd6-4d2c-b978-a55077005e59\" (UID: \"352ed02e-2cd6-4d2c-b978-a55077005e59\") " Oct 31 16:09:45.832621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832267 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tmp-dir" pod "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" (UID: "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.832621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832512 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-home\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-home\") pod \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\" (UID: \"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66\") " Oct 31 16:09:45.832621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832289 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-model-cache" pod "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" (UID: "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.832621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.832614 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-home" pod "352ed02e-2cd6-4d2c-b978-a55077005e59" (UID: "352ed02e-2cd6-4d2c-b978-a55077005e59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.833319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.833079 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-home" pod "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" (UID: "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.833319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.833205 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.833319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.833225 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.833319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.833239 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.833319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.833254 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.833319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.833297 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-tmp-dir" pod "352ed02e-2cd6-4d2c-b978-a55077005e59" (UID: "352ed02e-2cd6-4d2c-b978-a55077005e59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.833994 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.833323 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-model-cache" pod "352ed02e-2cd6-4d2c-b978-a55077005e59" (UID: "352ed02e-2cd6-4d2c-b978-a55077005e59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.835639 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.835612 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352ed02e-2cd6-4d2c-b978-a55077005e59-kube-api-access-7qpb7" pod "352ed02e-2cd6-4d2c-b978-a55077005e59" (UID: "352ed02e-2cd6-4d2c-b978-a55077005e59"). InnerVolumeSpecName "kube-api-access-7qpb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:45.836107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.836065 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-dshm" pod "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" (UID: "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.836219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.836148 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-dshm" pod "352ed02e-2cd6-4d2c-b978-a55077005e59" (UID: "352ed02e-2cd6-4d2c-b978-a55077005e59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.836357 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.836329 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352ed02e-2cd6-4d2c-b978-a55077005e59-tls-certs" pod "352ed02e-2cd6-4d2c-b978-a55077005e59" (UID: "352ed02e-2cd6-4d2c-b978-a55077005e59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:09:45.837001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.836979 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-kube-api-access-hmfv8" pod "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" (UID: "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66"). InnerVolumeSpecName "kube-api-access-hmfv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:45.837638 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.837617 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tls-certs" pod "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" (UID: "078f1c4f-9450-4770-ab7b-0ff3bfcd0f66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:09:45.934215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934182 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qpb7\" (UniqueName: \"kubernetes.io/projected/352ed02e-2cd6-4d2c-b978-a55077005e59-kube-api-access-7qpb7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.934215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934213 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.934215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934223 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.934496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934232 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.934496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934255 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.934496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934266 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352ed02e-2cd6-4d2c-b978-a55077005e59-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.934496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934275 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352ed02e-2cd6-4d2c-b978-a55077005e59-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.934496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:45.934284 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmfv8\" (UniqueName: \"kubernetes.io/projected/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66-kube-api-access-hmfv8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:46.626572 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.626534 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" Oct 31 16:09:46.627174 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.626539 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x" event={"ID":"352ed02e-2cd6-4d2c-b978-a55077005e59","Type":"ContainerDied","Data":"38a616039f888cc47360338fa8a07c3bb96dbd16f284f1d77d1924dcbe8c900a"} Oct 31 16:09:46.627174 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.626675 4865 scope.go:122] "RemoveContainer" containerID="7bd6756788eee17b8eddcebc6a90a74d77b2ed47e67f7f56f6b45c835a9fbbe1" Oct 31 16:09:46.628131 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.628020 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" event={"ID":"078f1c4f-9450-4770-ab7b-0ff3bfcd0f66","Type":"ContainerDied","Data":"5d85c959d44f1d3af55ba68074aa2660a1c99a42a0a857ff098aa34fb3d50f4c"} Oct 31 16:09:46.628131 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.628067 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m" Oct 31 16:09:46.637633 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.637589 4865 scope.go:122] "RemoveContainer" containerID="dc8cb268d8f785345c29df9afe6ad77ba43ccf6ce0ea62373175399bb269e656" Oct 31 16:09:46.792631 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.792591 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x"] Oct 31 16:09:46.843654 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.843623 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/simulator-w0-56bc96-kserve-6d9ff86c6b-7pw7x"] Oct 31 16:09:46.905797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.905766 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m"] Oct 31 16:09:46.909315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:46.909292 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-w0-56bc96-kserve-6fbc87f5f4-zd42m"] Oct 31 16:09:47.490023 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:09:47.489986 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:09:48.494913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:48.494880 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" path="/var/lib/kubelet/pods/078f1c4f-9450-4770-ab7b-0ff3bfcd0f66/volumes" Oct 31 16:09:48.495333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:48.495283 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" path="/var/lib/kubelet/pods/352ed02e-2cd6-4d2c-b978-a55077005e59/volumes" Oct 31 16:09:49.949579 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:49.949543 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:52.387652 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:52.387613 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:55.364510 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:55.364445 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:55.484413 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:55.484375 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:56.160332 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160299 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8"] Oct 31 16:09:56.160743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160727 4865 cpu_manager.go:414] "containerMap: removing container" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" Oct 31 16:09:56.160796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160745 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" Oct 31 16:09:56.160796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160762 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4663720c-0379-485b-9af9-7e90dcfc4fc7" containerName="payload-pre-processing" Oct 31 16:09:56.160796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160768 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4663720c-0379-485b-9af9-7e90dcfc4fc7" containerName="payload-pre-processing" Oct 31 16:09:56.160796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160780 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6ebd0091-6352-419d-a292-0fac0c76d495" containerName="revoke-keys" Oct 31 16:09:56.160796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160785 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebd0091-6352-419d-a292-0fac0c76d495" containerName="revoke-keys" Oct 31 16:09:56.160796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160791 4865 cpu_manager.go:414] "containerMap: removing container" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" Oct 31 16:09:56.160796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160796 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160806 4865 cpu_manager.go:414] "containerMap: removing container" podUID="5f0e1dbd-d3a0-4528-900c-529504a0fbed" containerName="payload-processing" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160810 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0e1dbd-d3a0-4528-900c-529504a0fbed" containerName="payload-processing" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160818 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" containerName="payload-pre-processing" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160823 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" containerName="payload-pre-processing" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160886 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="352ed02e-2cd6-4d2c-b978-a55077005e59" containerName="main" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160897 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4663720c-0379-485b-9af9-7e90dcfc4fc7" containerName="payload-pre-processing" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160911 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebd0091-6352-419d-a292-0fac0c76d495" containerName="revoke-keys" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160920 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="078f1c4f-9450-4770-ab7b-0ff3bfcd0f66" containerName="main" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160929 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0e1dbd-d3a0-4528-900c-529504a0fbed" containerName="payload-processing" Oct 31 16:09:56.161069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.160935 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e0e1ff-f21e-4725-b0c6-a523bb319651" containerName="payload-pre-processing" Oct 31 16:09:56.165190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.165171 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.218179 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.218152 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-56bc96-openshift-default-dockercfg-5qxbd\"" Oct 31 16:09:56.271394 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.271352 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8"] Oct 31 16:09:56.335589 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335561 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f070a-90ba-4061-a42f-a6238b74c645-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.335589 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335597 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.335828 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335623 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.335828 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335718 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f070a-90ba-4061-a42f-a6238b74c645-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.335828 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335782 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.335828 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335825 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.336044 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335847 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.336044 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335896 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.336044 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.335958 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggdb\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-kube-api-access-6ggdb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.436845 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.436747 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.436845 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.436795 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggdb\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-kube-api-access-6ggdb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.436845 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.436834 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f070a-90ba-4061-a42f-a6238b74c645-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.436859 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.436894 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.436926 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f070a-90ba-4061-a42f-a6238b74c645-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.436966 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.437015 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.437049 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.437231 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.437286 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.437517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.437505 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.438257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.437604 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f070a-90ba-4061-a42f-a6238b74c645-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.438257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.437650 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.439518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.439493 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.439619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.439534 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f070a-90ba-4061-a42f-a6238b74c645-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.495151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.495117 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.495274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.495194 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggdb\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-kube-api-access-6ggdb\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:56.778292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:56.778258 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:57.201546 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.201521 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8"] Oct 31 16:09:57.202673 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:09:57.202637 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937f070a_90ba_4061_a42f_a6238b74c645.slice/crio-c434d51cb169aea778f8c1d1f989ebc7650230efd62fe02faff3cdc79a2fe6d2 WatchSource:0}: Error finding container c434d51cb169aea778f8c1d1f989ebc7650230efd62fe02faff3cdc79a2fe6d2: Status 404 returned error can't find the container with id c434d51cb169aea778f8c1d1f989ebc7650230efd62fe02faff3cdc79a2fe6d2 Oct 31 16:09:57.204793 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.204758 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:09:57.204879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.204831 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:09:57.204879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.204862 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:09:57.691488 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.691443 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" event={"ID":"937f070a-90ba-4061-a42f-a6238b74c645","Type":"ContainerStarted","Data":"94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e"} Oct 31 16:09:57.691981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.691503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" event={"ID":"937f070a-90ba-4061-a42f-a6238b74c645","Type":"ContainerStarted","Data":"c434d51cb169aea778f8c1d1f989ebc7650230efd62fe02faff3cdc79a2fe6d2"} Oct 31 16:09:57.778995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.778965 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:09:57.860455 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:57.860402 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podStartSLOduration=1.860389013 podStartE2EDuration="1.860389013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:57.858956409 +0000 UTC m=+5043.478507820" watchObservedRunningTime="2026-10-31 16:09:57.860389013 +0000 UTC m=+5043.479940424" Oct 31 16:09:58.521205 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:58.521164 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:09:58.781157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:58.781071 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 31 16:09:58.781157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:58.781123 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 31 16:09:59.778991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:59.778958 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 31 16:09:59.779222 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:09:59.779024 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 31 16:10:00.046216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:00.046121 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55"] Oct 31 16:10:00.106495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:00.102391 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdabb34a-h8v55"] Oct 31 16:10:00.234279 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:00.234239 4865 scope.go:122] "RemoveContainer" containerID="8ff1bd947ab979d3c5dcc7c6f2caeb8de5774f621f0308c23506f1579d710c38" Oct 31 16:10:00.523494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:00.520171 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d35429-f86b-4a5c-9ec6-6b48c5c843bd" path="/var/lib/kubelet/pods/79d35429-f86b-4a5c-9ec6-6b48c5c843bd/volumes" Oct 31 16:10:00.778872 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:00.778779 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 31 16:10:00.778872 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:00.778844 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 31 16:10:01.779064 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:01.779026 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 31 16:10:01.779503 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:01.779093 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 31 16:10:02.779023 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:02.778985 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 31 16:10:02.779222 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:02.779046 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 31 16:10:03.778851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:03.778815 4865 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Oct 31 16:10:03.779053 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:03.778879 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Oct 31 16:10:05.547320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:05.547289 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:10:05.547841 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:05.547662 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:10:05.548295 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:05.548278 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:10:07.536741 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:07.536684 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:07.670997 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:07.670960 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:07.796393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:07.796310 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:09.811348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:09.811313 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:10.282333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:10.282290 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8"] Oct 31 16:10:10.282599 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:10.282569 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" containerID="cri-o://94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e" gracePeriod=30 Oct 31 16:10:10.315220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:10.315193 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:13.433849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:13.433810 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:13.566059 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:13.566019 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:13.615255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:13.615224 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2f7n9_cd1efbf8-d417-453e-a9e4-0be4b5b6d393/limitador/0.log" Oct 31 16:10:13.856819 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:13.856785 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:15.230831 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.230800 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz97q_200047c9-fccd-4826-b521-fce830acd38f/manager/0.log" Oct 31 16:10:15.364345 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.364313 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:15.532347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.532320 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:10:15.620711 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620673 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-certs\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.620908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620719 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-data\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.620908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620743 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-istio-token\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-istio-token\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.620908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620768 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937f070a-90ba-4061-a42f-a6238b74c645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f070a-90ba-4061-a42f-a6238b74c645-istiod-ca-cert\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.620908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620793 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/937f070a-90ba-4061-a42f-a6238b74c645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f070a-90ba-4061-a42f-a6238b74c645-istio-podinfo\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.620908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620856 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-credential-socket\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.620908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620888 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-socket\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.621274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620939 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-envoy\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.621274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.620990 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-kube-api-access-6ggdb\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-kube-api-access-6ggdb\") pod \"937f070a-90ba-4061-a42f-a6238b74c645\" (UID: \"937f070a-90ba-4061-a42f-a6238b74c645\") " Oct 31 16:10:15.621274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621022 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-certs" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.621274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621119 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-data" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.621516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621266 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-socket" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.621516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621283 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-credential-socket" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.621516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621389 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.621516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621409 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.621516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621423 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.621737 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.621526 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937f070a-90ba-4061-a42f-a6238b74c645-istiod-ca-cert" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:10:15.623343 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.623318 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-envoy" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.623459 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.623362 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-kube-api-access-6ggdb" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "kube-api-access-6ggdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:10:15.623718 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.623696 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-istio-token" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:10:15.623779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.623696 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/937f070a-90ba-4061-a42f-a6238b74c645-istio-podinfo" pod "937f070a-90ba-4061-a42f-a6238b74c645" (UID: "937f070a-90ba-4061-a42f-a6238b74c645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:10:15.722090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.722051 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.722090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.722093 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f070a-90ba-4061-a42f-a6238b74c645-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.722333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.722111 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggdb\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-kube-api-access-6ggdb\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.722333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.722126 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f070a-90ba-4061-a42f-a6238b74c645-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.722333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.722141 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f070a-90ba-4061-a42f-a6238b74c645-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.722333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.722155 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f070a-90ba-4061-a42f-a6238b74c645-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.729291 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.729263 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:15.802637 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.802546 4865 generic.go:356] "Generic (PLEG): container finished" podID="937f070a-90ba-4061-a42f-a6238b74c645" containerID="94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e" exitCode=0 Oct 31 16:10:15.802637 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.802601 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" Oct 31 16:10:15.802871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.802631 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" event={"ID":"937f070a-90ba-4061-a42f-a6238b74c645","Type":"ContainerDied","Data":"94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e"} Oct 31 16:10:15.802871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.802686 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8" event={"ID":"937f070a-90ba-4061-a42f-a6238b74c645","Type":"ContainerDied","Data":"c434d51cb169aea778f8c1d1f989ebc7650230efd62fe02faff3cdc79a2fe6d2"} Oct 31 16:10:15.802871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.802705 4865 scope.go:122] "RemoveContainer" containerID="94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e" Oct 31 16:10:15.813373 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.813355 4865 scope.go:122] "RemoveContainer" containerID="94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e" Oct 31 16:10:15.813694 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:10:15.813675 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e\": container with ID starting with 94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e not found: ID does not exist" containerID="94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e" Oct 31 16:10:15.813765 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.813703 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e"} err="failed to get container status \"94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e\": rpc error: code = NotFound desc = could not find container \"94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e\": container with ID starting with 94fab5065b1cc8e6b791bbab394d96bb7beac004c3114e621ebec08f4e92826e not found: ID does not exist" Oct 31 16:10:15.978205 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:15.978172 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8"] Oct 31 16:10:16.028966 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:16.028923 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-vlbc8"] Oct 31 16:10:16.055401 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:16.055322 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf"] Oct 31 16:10:16.060067 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:16.060042 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06a99e9-vpkgf"] Oct 31 16:10:16.495214 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:16.495176 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937f070a-90ba-4061-a42f-a6238b74c645" path="/var/lib/kubelet/pods/937f070a-90ba-4061-a42f-a6238b74c645/volumes" Oct 31 16:10:16.495657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:16.495605 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02b3b7a-88d8-4325-a130-0965a02c1109" path="/var/lib/kubelet/pods/c02b3b7a-88d8-4325-a130-0965a02c1109/volumes" Oct 31 16:10:17.490182 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:10:17.490145 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:10:17.936282 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:17.936198 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5vzk_4e96a5f7-706c-4405-be53-e4362dc7677d/manager/0.log" Oct 31 16:10:22.798550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:22.798507 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:23.783483 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.783432 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx"] Oct 31 16:10:23.783913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.783891 4865 cpu_manager.go:414] "containerMap: removing container" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" Oct 31 16:10:23.783913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.783910 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" Oct 31 16:10:23.784138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.783996 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="937f070a-90ba-4061-a42f-a6238b74c645" containerName="istio-proxy" Oct 31 16:10:23.788296 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.788278 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.841177 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.841148 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9e4bd6cf-openshift-default-dockercfg-b58z4\"" Oct 31 16:10:23.874549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.874511 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:23.894224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.894189 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx"] Oct 31 16:10:23.900264 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900229 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fwb9\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-kube-api-access-9fwb9\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900286 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900400 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900435 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900453 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900549 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900587 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900635 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:23.900779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:23.900680 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.001987 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.001951 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.001987 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.001989 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002010 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002025 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002043 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002060 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002081 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002132 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002205 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fwb9\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-kube-api-access-9fwb9\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002436 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002518 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002592 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002648 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.002921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.002903 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.004714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.004691 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.004833 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.004812 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.055303 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.055208 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:24.059414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.059386 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.059591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.059573 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fwb9\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-kube-api-access-9fwb9\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.101444 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.101410 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:24.525762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.525734 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx"] Oct 31 16:10:24.527699 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:10:24.527670 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode263ce0c_7c4f_4480_aefa_c4021fff9d16.slice/crio-ea6d0627128bb4af6b2c22856ad176e141877c7cb6ec440f81551048450a1135 WatchSource:0}: Error finding container ea6d0627128bb4af6b2c22856ad176e141877c7cb6ec440f81551048450a1135: Status 404 returned error can't find the container with id ea6d0627128bb4af6b2c22856ad176e141877c7cb6ec440f81551048450a1135 Oct 31 16:10:24.529822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.529784 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:10:24.529928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.529858 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:10:24.529928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.529887 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:10:24.853122 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.853026 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" event={"ID":"e263ce0c-7c4f-4480-aefa-c4021fff9d16","Type":"ContainerStarted","Data":"8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62"} Oct 31 16:10:24.853122 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.853088 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" event={"ID":"e263ce0c-7c4f-4480-aefa-c4021fff9d16","Type":"ContainerStarted","Data":"ea6d0627128bb4af6b2c22856ad176e141877c7cb6ec440f81551048450a1135"} Oct 31 16:10:24.972039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:24.971978 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" podStartSLOduration=1.971964039 podStartE2EDuration="1.971964039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:10:24.969894503 +0000 UTC m=+5070.589445915" watchObservedRunningTime="2026-10-31 16:10:24.971964039 +0000 UTC m=+5070.591515451" Oct 31 16:10:25.102338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:25.102293 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:25.107463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:25.107394 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:25.859486 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:25.859446 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:25.860500 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:25.860458 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:26.081967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:26.081926 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm"] Oct 31 16:10:26.083981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:26.083952 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ee7413-8twnm"] Oct 31 16:10:26.495113 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:26.495070 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1ec719-741b-4bd3-9caf-8520bb32c7c9" path="/var/lib/kubelet/pods/bc1ec719-741b-4bd3-9caf-8520bb32c7c9/volumes" Oct 31 16:10:27.034185 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:27.034142 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:27.209764 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:27.209729 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:27.345325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:27.345241 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:30.243674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.243639 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:30.412677 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.412646 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:30.655001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.654918 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j"] Oct 31 16:10:30.659512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.659491 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:30.715766 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.715727 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j"] Oct 31 16:10:30.763098 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.763033 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9e4bd6cf\"" Oct 31 16:10:30.763312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.763109 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9e4bd6cf-dockercfg-b67nw\"" Oct 31 16:10:30.763312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.763254 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3008474a-cc15-4493-a59b-363e421502cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:30.763424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.763311 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtt8n\" (UniqueName: \"kubernetes.io/projected/3008474a-cc15-4493-a59b-363e421502cb-kube-api-access-wtt8n\") pod \"payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:30.816644 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.816607 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k"] Oct 31 16:10:30.820153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.820130 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:30.863879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.863841 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa687d12-2a1d-4d02-9b0b-abb058edc160-plugins-config-volume\") pod \"payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:30.863879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.863887 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5dg\" (UniqueName: \"kubernetes.io/projected/fa687d12-2a1d-4d02-9b0b-abb058edc160-kube-api-access-2j5dg\") pod \"payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:30.864120 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.863994 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3008474a-cc15-4493-a59b-363e421502cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:30.864120 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.864051 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtt8n\" (UniqueName: \"kubernetes.io/projected/3008474a-cc15-4493-a59b-363e421502cb-kube-api-access-wtt8n\") pod \"payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:30.864684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.864664 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3008474a-cc15-4493-a59b-363e421502cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:30.874728 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.874692 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k"] Oct 31 16:10:30.922104 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.922024 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtt8n\" (UniqueName: \"kubernetes.io/projected/3008474a-cc15-4493-a59b-363e421502cb-kube-api-access-wtt8n\") pod \"payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:30.964984 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.964943 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa687d12-2a1d-4d02-9b0b-abb058edc160-plugins-config-volume\") pod \"payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:30.964984 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.964991 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5dg\" (UniqueName: \"kubernetes.io/projected/fa687d12-2a1d-4d02-9b0b-abb058edc160-kube-api-access-2j5dg\") pod \"payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:30.965714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.965691 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa687d12-2a1d-4d02-9b0b-abb058edc160-plugins-config-volume\") pod \"payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:30.973822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:30.973793 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:31.023019 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.022979 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5dg\" (UniqueName: \"kubernetes.io/projected/fa687d12-2a1d-4d02-9b0b-abb058edc160-kube-api-access-2j5dg\") pod \"payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:31.133132 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.133091 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:31.344583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.344559 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j"] Oct 31 16:10:31.345940 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:10:31.345912 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3008474a_cc15_4493_a59b_363e421502cb.slice/crio-ef4390a8461211d1c5b67aa8e07a684ab3b01828a3327d80f99dcfd0fa5ce5c4 WatchSource:0}: Error finding container ef4390a8461211d1c5b67aa8e07a684ab3b01828a3327d80f99dcfd0fa5ce5c4: Status 404 returned error can't find the container with id ef4390a8461211d1c5b67aa8e07a684ab3b01828a3327d80f99dcfd0fa5ce5c4 Oct 31 16:10:31.504727 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.504704 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k"] Oct 31 16:10:31.506084 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:10:31.506047 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa687d12_2a1d_4d02_9b0b_abb058edc160.slice/crio-e0487954be3de66d30cbc791d44c8fe0cda01992883c40ce171ad1eb67d75d57 WatchSource:0}: Error finding container e0487954be3de66d30cbc791d44c8fe0cda01992883c40ce171ad1eb67d75d57: Status 404 returned error can't find the container with id e0487954be3de66d30cbc791d44c8fe0cda01992883c40ce171ad1eb67d75d57 Oct 31 16:10:31.899597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.899457 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" event={"ID":"3008474a-cc15-4493-a59b-363e421502cb","Type":"ContainerStarted","Data":"396512a9452a9b0ac9834bea1aa6ddbdcb97838dc2b418bae65649681b77855f"} Oct 31 16:10:31.899597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.899557 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" event={"ID":"3008474a-cc15-4493-a59b-363e421502cb","Type":"ContainerStarted","Data":"ef4390a8461211d1c5b67aa8e07a684ab3b01828a3327d80f99dcfd0fa5ce5c4"} Oct 31 16:10:31.899829 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.899649 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:31.901178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.901146 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" event={"ID":"fa687d12-2a1d-4d02-9b0b-abb058edc160","Type":"ContainerStarted","Data":"4360a034a24f379bf7a0549b7d7755efe1f6adcc065037dde1aa3cb260a18896"} Oct 31 16:10:31.901308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.901199 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" event={"ID":"fa687d12-2a1d-4d02-9b0b-abb058edc160","Type":"ContainerStarted","Data":"e0487954be3de66d30cbc791d44c8fe0cda01992883c40ce171ad1eb67d75d57"} Oct 31 16:10:31.901308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:31.901221 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:32.014957 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:32.014900 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" podStartSLOduration=2.014885288 podStartE2EDuration="2.014885288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:10:32.013818783 +0000 UTC m=+5077.633370198" watchObservedRunningTime="2026-10-31 16:10:32.014885288 +0000 UTC m=+5077.634436698" Oct 31 16:10:32.128761 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:32.128703 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" podStartSLOduration=2.128686596 podStartE2EDuration="2.128686596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:10:32.127948635 +0000 UTC m=+5077.747500047" watchObservedRunningTime="2026-10-31 16:10:32.128686596 +0000 UTC m=+5077.748238020" Oct 31 16:10:33.078882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:33.078844 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:33.227524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:33.227449 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:33.400917 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:33.400828 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:37.046823 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:37.046732 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd"] Oct 31 16:10:37.080714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:37.080679 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e1a9055-n8twd"] Oct 31 16:10:37.888288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:37.888250 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:38.495364 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:38.495310 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba618bb-ec1f-4748-a68c-74002b195357" path="/var/lib/kubelet/pods/3ba618bb-ec1f-4748-a68c-74002b195357/volumes" Oct 31 16:10:42.683517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:42.683460 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:42.919729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:42.919692 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:10:42.920870 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:42.920848 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:10:47.394660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:47.394621 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:47.491443 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:10:47.491400 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:10:49.996522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:49.996483 4865 generic.go:356] "Generic (PLEG): container finished" podID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerID="8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62" exitCode=0 Oct 31 16:10:49.997018 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:49.996499 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" event={"ID":"e263ce0c-7c4f-4480-aefa-c4021fff9d16","Type":"ContainerDied","Data":"8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62"} Oct 31 16:10:49.997018 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:49.996921 4865 scope.go:122] "RemoveContainer" containerID="8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62" Oct 31 16:10:49.998188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:49.998156 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:10:49.998289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:49.998227 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:10:49.998289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:49.998265 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:10:51.022695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:51.022652 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" event={"ID":"e263ce0c-7c4f-4480-aefa-c4021fff9d16","Type":"ContainerStarted","Data":"a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4"} Oct 31 16:10:51.111493 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:51.110938 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:51.118498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:51.117599 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 31 16:10:51.118498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:51.117687 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 31 16:10:52.113494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:52.112578 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 31 16:10:52.113494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:52.112658 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 31 16:10:52.413553 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:52.409204 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:53.110970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:53.110924 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 31 16:10:53.111315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:53.111281 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 31 16:10:53.633508 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:10:53.633446 4865 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-60.us-west-1.compute.internal?timeout=10s\": context deadline exceeded" Oct 31 16:10:54.107007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:54.106962 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 31 16:10:54.107321 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:54.107291 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 31 16:10:55.118648 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:55.118597 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Oct 31 16:10:55.119150 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:55.118669 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Oct 31 16:10:55.859630 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:55.859566 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:56.163984 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:56.163893 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:56.165259 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:56.165234 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:10:56.289833 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:56.289802 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:10:56.298403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:56.298377 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:10:57.511604 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:57.511567 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:58.749666 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:58.749634 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:10:58.758361 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:10:58.758342 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:11:00.281251 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:00.281214 4865 scope.go:122] "RemoveContainer" containerID="10196b61a26f48d3a87eabab88beb7a161d6970fef0277564ae5b756e3fec4b9" Oct 31 16:11:00.290574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:00.290554 4865 scope.go:122] "RemoveContainer" containerID="067ba4945a3fce56b21b84b552db7777da82212ea0343e58abe42ef0c11ffb5e" Oct 31 16:11:02.273644 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:02.273580 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:03.634141 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:03.634091 4865 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-60.us-west-1.compute.internal?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:11:06.999134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:06.999088 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:11.727173 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:11.727139 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:13.635005 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:13.634957 4865 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-60.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:11:14.203045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:14.203000 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:15.724392 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:15.724358 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:16.420355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:16.420321 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:17.388976 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:17.388918 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:17.492622 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:17.492588 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:11:21.097832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:21.097796 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079460 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079510 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079526 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8njhh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079540 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079549 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079559 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079569 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079542 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.079574 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079579 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079595 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079607 4865 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079580 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079621 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9e4bd6cf-dockercfg-b67nw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079634 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079643 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079648 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079654 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079666 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079671 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079677 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079707 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079711 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079740 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whbtk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079751 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079755 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079765 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xtrmf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079797 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079803 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079805 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079818 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079829 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079836 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079842 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079849 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079859 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tgszw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079874 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079882 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079887 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079900 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mg2kx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079912 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079923 4865 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079928 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079945 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.079976 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080005 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080026 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080028 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080042 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080053 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080064 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080079 4865 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080080 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-ait-9e4bd6cf-openshift-default-dockercfg-b58z4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080101 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080105 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080113 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080124 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080132 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080139 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080155 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080169 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080173 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080171 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080175 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080198 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080183 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080194 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080210 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080219 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080225 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080215 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sffft\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080250 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080259 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080264 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080263 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080283 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fzc6l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080287 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080302 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080305 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9e4bd6cf\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080324 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080337 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080337 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.080285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080347 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080363 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080378 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080376 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zxl7k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080402 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080406 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080402 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080404 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080378 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080418 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsph8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080430 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080427 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080434 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080445 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080464 4865 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080495 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080502 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080494 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:22.080379 4865 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/events/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx.18e3a91e5edf7ba2\": http2: client connection lost" event="&Event{ObjectMeta:{e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx.18e3a91e5edf7ba2 openshift-ingress 148571 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-ingress,Name:e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx,UID:e263ce0c-7c4f-4480-aefa-c4021fff9d16,APIVersion:v1,ResourceVersion:148545,FieldPath:spec.containers{istio-proxy},},Reason:Pulled,Message:Container image \"registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55\" already present on machine and can be accessed by the pod,Source:EventSource{Component:kubelet,Host:ip-10-0-131-60.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 16:10:24 +0000 UTC,LastTimestamp:2026-10-31 16:10:49.997825107 +0000 UTC m=+5095.617376519,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-60.us-west-1.compute.internal,}" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080508 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080464 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gsm2m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080513 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080543 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080511 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gm7jd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080498 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080557 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080557 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080556 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080584 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080591 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080523 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080557 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080609 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080584 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080616 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080613 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080527 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080594 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080641 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dm4bk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080639 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080657 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080659 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080642 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-m92fm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080644 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080670 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080674 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t2tz4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080679 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080700 4865 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080705 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080695 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080716 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080721 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xknrx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080677 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080732 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080736 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:22.080704 4865 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-60.us-west-1.compute.internal?timeout=10s\": http2: client connection lost" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080723 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080732 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080719 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080707 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.083154 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.080745 4865 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:11:22.945071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.945034 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gm7jd\"" Oct 31 16:11:22.950061 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.950038 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 16:11:22.994857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.994821 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 16:11:22.995042 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.994886 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tgszw\"" Oct 31 16:11:22.995042 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.994901 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 16:11:22.999826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.999804 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 16:11:22.999893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:22.999804 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fzc6l\"" Oct 31 16:11:23.009672 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.009645 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 16:11:23.044542 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.044502 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whbtk\"" Oct 31 16:11:23.044735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.044571 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 16:11:23.044735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.044583 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.044735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.044667 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 16:11:23.044735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.044712 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.044927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.044778 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 16:11:23.044927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.044785 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" Oct 31 16:11:23.049635 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.049622 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 16:11:23.059417 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.059400 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 16:11:23.094118 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094088 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 16:11:23.094118 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094111 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" Oct 31 16:11:23.094348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094203 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 16:11:23.094348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094228 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" Oct 31 16:11:23.094348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094315 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.094348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094332 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" Oct 31 16:11:23.094626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094380 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 16:11:23.094626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094405 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 16:11:23.094626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094445 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" Oct 31 16:11:23.094626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094447 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 16:11:23.094626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094492 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 16:11:23.094626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094462 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.094626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.094517 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xknrx\"" Oct 31 16:11:23.108481 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.108457 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" Oct 31 16:11:23.109186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.109172 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.114002 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.113982 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 16:11:23.124203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.124180 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.130210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.130191 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.133340 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.133319 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.143757 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.143735 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 16:11:23.146071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.146049 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.149237 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.149216 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 16:11:23.151341 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.151326 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.155693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.155676 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 16:11:23.158745 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.158730 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 16:11:23.160562 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.160542 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.160918 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.160904 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 16:11:23.163213 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.163196 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9e4bd6cf-dockercfg-b67nw\"" Oct 31 16:11:23.183172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.183150 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 16:11:23.190086 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.190069 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gsm2m\"" Oct 31 16:11:23.202794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.197636 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 16:11:23.202794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.197970 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 16:11:23.205042 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.205012 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.209020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.208997 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 16:11:23.210779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.210762 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\"" Oct 31 16:11:23.224796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.224768 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 16:11:23.230809 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.230783 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.236434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.236412 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" Oct 31 16:11:23.236901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.236883 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.237029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.237008 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9e4bd6cf-openshift-default-dockercfg-b58z4\"" Oct 31 16:11:23.243235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.243216 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 16:11:23.243235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.243220 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 16:11:23.243891 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.243875 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 16:11:23.255344 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.255323 4865 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:11:23.268308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.268278 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 16:11:23.268308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.268295 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.270403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.270383 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.271351 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.271334 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sffft\"" Oct 31 16:11:23.284336 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.284311 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 16:11:23.291464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.291447 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 16:11:23.312002 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.311972 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 16:11:23.312199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.312074 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 16:11:23.313902 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.313882 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.318086 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.318067 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 16:11:23.321642 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.321630 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsph8\"" Oct 31 16:11:23.323558 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.323532 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 16:11:23.328237 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.328220 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" Oct 31 16:11:23.329939 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.329923 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 16:11:23.341821 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.341802 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.349140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.349124 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dm4bk\"" Oct 31 16:11:23.356210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.356193 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.360346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.360333 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.363722 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.363700 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 16:11:23.363834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.363820 4865 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:11:23.368285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.368268 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.373115 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.373093 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 16:11:23.375744 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.375731 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\"" Oct 31 16:11:23.381566 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.381552 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 16:11:23.385454 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.385437 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 16:11:23.396481 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.396456 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 16:11:23.398550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.398532 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.398663 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.398649 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mg2kx\"" Oct 31 16:11:23.400507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.400492 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.410683 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.410666 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" Oct 31 16:11:23.413070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.413055 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 16:11:23.417140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.417126 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 16:11:23.429262 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.429240 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" Oct 31 16:11:23.431553 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.431539 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.434250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.434229 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 16:11:23.460702 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.460683 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 16:11:23.461817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.461802 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" Oct 31 16:11:23.465649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.465636 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t2tz4\"" Oct 31 16:11:23.471547 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.471531 4865 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:11:23.483782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.483762 4865 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:11:23.484522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.484506 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.487873 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.487855 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.489994 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.489972 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 16:11:23.493160 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.493140 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 16:11:23.496159 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.496136 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.498297 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.498273 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9e4bd6cf\"" Oct 31 16:11:23.517310 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.517286 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" Oct 31 16:11:23.518300 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.518279 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 16:11:23.521514 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.521497 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" Oct 31 16:11:23.524027 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.524008 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.529328 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.529258 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.531424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.531404 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.532399 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.532379 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.549082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.549056 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 16:11:23.564178 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.564151 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m92fm\"" Oct 31 16:11:23.571809 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.571788 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.574395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.574372 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" Oct 31 16:11:23.578633 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.578614 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.579458 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.579442 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.581826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.581811 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.598818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.598793 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xtrmf\"" Oct 31 16:11:23.599867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.599852 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.603180 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.603162 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" Oct 31 16:11:23.604171 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.604156 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.628308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.628288 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" Oct 31 16:11:23.628402 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.628389 4865 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 16:11:23.631639 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.631618 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1c8dc8-q6ld5","odh-ai-gateway-infra/maas-api-revoke-sub-214aae44-ppd9z","odh-ai-gateway-infra/maas-api-revoke-sub-5ff2e88e-wxqll"] Oct 31 16:11:23.631704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.631665 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz","openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87"] Oct 31 16:11:23.634886 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.634868 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.637787 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.637768 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:11:23.638047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.638025 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.645032 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.645012 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 16:11:23.645143 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.645047 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 16:11:23.648376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.648356 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.663066 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.663045 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 16:11:23.665087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.665071 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.666152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.666134 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.671436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.671419 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 16:11:23.680612 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.680595 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zxl7k\"" Oct 31 16:11:23.683431 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.683417 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 16:11:23.686837 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.686824 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:11:23.686881 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.686861 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.686915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.686889 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.686957 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.686939 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:11:23.687081 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.687067 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 16:11:23.695287 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.695271 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8njhh\"" Oct 31 16:11:23.708756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.708732 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.708854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.708768 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.708854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.708793 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlws4\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-kube-api-access-tlws4\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.708938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.708859 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.708978 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.708932 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.708978 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.708954 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.709046 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.708993 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6339af6-924c-4eac-bb95-cae4777d25a0-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.709141 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.709127 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tqg4\" (UniqueName: \"kubernetes.io/projected/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-kube-api-access-4tqg4\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.709196 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.709179 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.709304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.709292 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.709342 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.709321 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.709381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.709345 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.710065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.710044 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 16:11:23.723935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.723877 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:11:23.810094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810059 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810102 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810294 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlws4\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-kube-api-access-tlws4\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810313 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810336 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.810557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810356 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810389 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6339af6-924c-4eac-bb95-cae4777d25a0-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810443 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tqg4\" (UniqueName: \"kubernetes.io/projected/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-kube-api-access-4tqg4\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.810557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810520 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810556 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.810860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810799 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.810860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.810843 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.811151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.811123 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.811243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.811126 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.811243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.811160 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.811337 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.811325 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.811384 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.811335 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.811826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.811804 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6339af6-924c-4eac-bb95-cae4777d25a0-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.813007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.812987 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.813682 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.813658 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.813775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.813709 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.866760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.866729 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tqg4\" (UniqueName: \"kubernetes.io/projected/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-kube-api-access-4tqg4\") pod \"maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.867323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.867298 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.867691 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.867673 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlws4\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-kube-api-access-tlws4\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:23.949207 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.949180 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:23.955261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:23.955230 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:24.325859 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.325779 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz"] Oct 31 16:11:24.365850 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:11:24.365823 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode967ba6d_6b9b_4208_b47b_7de9c1d7e32c.slice/crio-9d0c6eb8578cbc8fe2defc4328a8f5df258603fb4f681789a51180392cb45fe7 WatchSource:0}: Error finding container 9d0c6eb8578cbc8fe2defc4328a8f5df258603fb4f681789a51180392cb45fe7: Status 404 returned error can't find the container with id 9d0c6eb8578cbc8fe2defc4328a8f5df258603fb4f681789a51180392cb45fe7 Oct 31 16:11:24.378822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.378795 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87"] Oct 31 16:11:24.379039 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:11:24.378988 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6339af6_924c_4eac_bb95_cae4777d25a0.slice/crio-3a5608062742adc7240b78c567ca5fb5d4d5e8c7d276bddb09071e9bd720087d WatchSource:0}: Error finding container 3a5608062742adc7240b78c567ca5fb5d4d5e8c7d276bddb09071e9bd720087d: Status 404 returned error can't find the container with id 3a5608062742adc7240b78c567ca5fb5d4d5e8c7d276bddb09071e9bd720087d Oct 31 16:11:24.381155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.381123 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:11:24.381252 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.381190 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:11:24.381252 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.381224 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:11:24.496923 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.496881 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075e66dd-8540-472c-9636-d924b5c86b13" path="/var/lib/kubelet/pods/075e66dd-8540-472c-9636-d924b5c86b13/volumes" Oct 31 16:11:24.497610 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.497580 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec13819-1167-4dd6-9314-117272a8ad22" path="/var/lib/kubelet/pods/4ec13819-1167-4dd6-9314-117272a8ad22/volumes" Oct 31 16:11:24.498277 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:24.498257 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f789656-052e-4c05-acef-cca7eaf9d108" path="/var/lib/kubelet/pods/9f789656-052e-4c05-acef-cca7eaf9d108/volumes" Oct 31 16:11:25.225132 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.225090 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" event={"ID":"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c","Type":"ContainerStarted","Data":"8c4ca0cb203bcf1033ec7097bd4c0622738b03140c93068a7a290aef05ff42cb"} Oct 31 16:11:25.225642 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.225154 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" event={"ID":"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c","Type":"ContainerStarted","Data":"9d0c6eb8578cbc8fe2defc4328a8f5df258603fb4f681789a51180392cb45fe7"} Oct 31 16:11:25.226806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.226778 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" event={"ID":"d6339af6-924c-4eac-bb95-cae4777d25a0","Type":"ContainerStarted","Data":"55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe"} Oct 31 16:11:25.226952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.226829 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" event={"ID":"d6339af6-924c-4eac-bb95-cae4777d25a0","Type":"ContainerStarted","Data":"3a5608062742adc7240b78c567ca5fb5d4d5e8c7d276bddb09071e9bd720087d"} Oct 31 16:11:25.340687 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.340632 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" podStartSLOduration=22.340618933000002 podStartE2EDuration="22.340618933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:25.339294202 +0000 UTC m=+5130.958845614" watchObservedRunningTime="2026-10-31 16:11:25.340618933 +0000 UTC m=+5130.960170343" Oct 31 16:11:25.455231 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.455171 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podStartSLOduration=16.455150524 podStartE2EDuration="16.455150524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:25.453089622 +0000 UTC m=+5131.072641064" watchObservedRunningTime="2026-10-31 16:11:25.455150524 +0000 UTC m=+5131.074701936" Oct 31 16:11:25.790194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.790158 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:25.956814 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.956776 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:25.958232 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.958209 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body= Oct 31 16:11:25.958317 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:25.958283 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" Oct 31 16:11:26.234148 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:26.234112 4865 generic.go:356] "Generic (PLEG): container finished" podID="e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" containerID="8c4ca0cb203bcf1033ec7097bd4c0622738b03140c93068a7a290aef05ff42cb" exitCode=0 Oct 31 16:11:26.234695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:26.234198 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" event={"ID":"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c","Type":"ContainerDied","Data":"8c4ca0cb203bcf1033ec7097bd4c0622738b03140c93068a7a290aef05ff42cb"} Oct 31 16:11:26.956684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:26.956641 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body= Oct 31 16:11:26.956880 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:26.956705 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" Oct 31 16:11:27.426637 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.426603 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:27.548797 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.548724 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-xks-ca\") pod \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " Oct 31 16:11:27.548952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.548873 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-openshift-ca\") pod \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " Oct 31 16:11:27.548952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.548895 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-kube-api-access-4tqg4\" (UniqueName: \"kubernetes.io/projected/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-kube-api-access-4tqg4\") pod \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\" (UID: \"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c\") " Oct 31 16:11:27.549319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.549292 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-openshift-ca" pod "e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" (UID: "e967ba6d-6b9b-4208-b47b-7de9c1d7e32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:27.551275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.551242 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-xks-ca" pod "e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" (UID: "e967ba6d-6b9b-4208-b47b-7de9c1d7e32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:11:27.552625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.552604 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-kube-api-access-4tqg4" pod "e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" (UID: "e967ba6d-6b9b-4208-b47b-7de9c1d7e32c"). InnerVolumeSpecName "kube-api-access-4tqg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:27.650195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.650162 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:27.650195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.650191 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tqg4\" (UniqueName: \"kubernetes.io/projected/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-kube-api-access-4tqg4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:27.650195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.650201 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:27.956751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.956663 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body= Oct 31 16:11:27.956751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:27.956723 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" Oct 31 16:11:28.047444 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.047407 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh"] Oct 31 16:11:28.121998 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.121959 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161d3702-slxgh"] Oct 31 16:11:28.246727 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.246695 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" Oct 31 16:11:28.246906 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.246727 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz" event={"ID":"e967ba6d-6b9b-4208-b47b-7de9c1d7e32c","Type":"ContainerDied","Data":"9d0c6eb8578cbc8fe2defc4328a8f5df258603fb4f681789a51180392cb45fe7"} Oct 31 16:11:28.246906 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.246776 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d0c6eb8578cbc8fe2defc4328a8f5df258603fb4f681789a51180392cb45fe7" Oct 31 16:11:28.504561 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.496354 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc91dd7-eae5-4a7a-bb8b-c3255957c449" path="/var/lib/kubelet/pods/0cc91dd7-eae5-4a7a-bb8b-c3255957c449/volumes" Oct 31 16:11:28.956874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.956789 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body= Oct 31 16:11:28.956874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:28.956840 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" Oct 31 16:11:29.956648 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:29.956603 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" start-of-body= Oct 31 16:11:29.957078 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:29.956700 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.145:15021/healthz/ready\": dial tcp 10.132.1.145:15021: connect: connection refused" Oct 31 16:11:30.505161 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:30.505125 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:30.960059 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:30.960027 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:30.960596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:30.960406 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:30.961136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:30.961119 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:35.076981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:35.076945 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k"] Oct 31 16:11:35.181603 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:35.181567 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j"] Oct 31 16:11:35.205182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:35.205153 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:35.425136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:35.425046 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:35.549862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:35.549791 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:39.366291 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.366254 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:39.482605 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.482564 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb"] Oct 31 16:11:39.483054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.483040 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" containerName="revoke-keys" Oct 31 16:11:39.483119 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.483060 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" containerName="revoke-keys" Oct 31 16:11:39.483166 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.483129 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" containerName="revoke-keys" Oct 31 16:11:39.486216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.486189 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559163 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559125 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559163 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559170 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559196 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9nv\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-kube-api-access-pm9nv\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559311 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559354 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559383 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559400 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36531d0f-fb84-402e-a413-a5f1acd95106-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559418 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36531d0f-fb84-402e-a413-a5f1acd95106-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.559697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.559521 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.578259 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.578220 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb"] Oct 31 16:11:39.660153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660070 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660108 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36531d0f-fb84-402e-a413-a5f1acd95106-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660131 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36531d0f-fb84-402e-a413-a5f1acd95106-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660150 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660204 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660225 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660249 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9nv\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-kube-api-access-pm9nv\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660275 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660314 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660611 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660672 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.660876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660859 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.661022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660909 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.661022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.660964 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36531d0f-fb84-402e-a413-a5f1acd95106-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.662765 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.662746 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.662990 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.662974 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36531d0f-fb84-402e-a413-a5f1acd95106-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.717035 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.717003 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.717293 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.717257 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9nv\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-kube-api-access-pm9nv\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:39.798732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:39.798681 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:40.077275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.077213 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" podUID="fa687d12-2a1d-4d02-9b0b-abb058edc160" containerName="payload-processing" containerID="cri-o://4360a034a24f379bf7a0549b7d7755efe1f6adcc065037dde1aa3cb260a18896" gracePeriod=25 Oct 31 16:11:40.182045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.181969 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" podUID="3008474a-cc15-4493-a59b-363e421502cb" containerName="payload-pre-processing" containerID="cri-o://396512a9452a9b0ac9834bea1aa6ddbdcb97838dc2b418bae65649681b77855f" gracePeriod=25 Oct 31 16:11:40.225399 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.225354 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb"] Oct 31 16:11:40.256851 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:11:40.256820 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36531d0f_fb84_402e_a413_a5f1acd95106.slice/crio-2d596ae8409f82eb8abbaba173831addbd91fdb35739f87f5d90b56106a85144 WatchSource:0}: Error finding container 2d596ae8409f82eb8abbaba173831addbd91fdb35739f87f5d90b56106a85144: Status 404 returned error can't find the container with id 2d596ae8409f82eb8abbaba173831addbd91fdb35739f87f5d90b56106a85144 Oct 31 16:11:40.288424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.288382 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:11:40.288605 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.288579 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:11:40.288690 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.288646 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:11:40.313412 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.313377 4865 generic.go:356] "Generic (PLEG): container finished" podID="3008474a-cc15-4493-a59b-363e421502cb" containerID="396512a9452a9b0ac9834bea1aa6ddbdcb97838dc2b418bae65649681b77855f" exitCode=0 Oct 31 16:11:40.313628 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.313444 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" event={"ID":"3008474a-cc15-4493-a59b-363e421502cb","Type":"ContainerDied","Data":"396512a9452a9b0ac9834bea1aa6ddbdcb97838dc2b418bae65649681b77855f"} Oct 31 16:11:40.315742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.315709 4865 generic.go:356] "Generic (PLEG): container finished" podID="fa687d12-2a1d-4d02-9b0b-abb058edc160" containerID="4360a034a24f379bf7a0549b7d7755efe1f6adcc065037dde1aa3cb260a18896" exitCode=0 Oct 31 16:11:40.315874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.315764 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" event={"ID":"fa687d12-2a1d-4d02-9b0b-abb058edc160","Type":"ContainerDied","Data":"4360a034a24f379bf7a0549b7d7755efe1f6adcc065037dde1aa3cb260a18896"} Oct 31 16:11:40.323707 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.318104 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" event={"ID":"36531d0f-fb84-402e-a413-a5f1acd95106","Type":"ContainerStarted","Data":"2d596ae8409f82eb8abbaba173831addbd91fdb35739f87f5d90b56106a85144"} Oct 31 16:11:40.326832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.326812 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:11:40.448845 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.448814 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:11:40.469323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.469290 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa687d12-2a1d-4d02-9b0b-abb058edc160-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa687d12-2a1d-4d02-9b0b-abb058edc160-plugins-config-volume\") pod \"fa687d12-2a1d-4d02-9b0b-abb058edc160\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " Oct 31 16:11:40.469554 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.469425 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa687d12-2a1d-4d02-9b0b-abb058edc160-kube-api-access-2j5dg\" (UniqueName: \"kubernetes.io/projected/fa687d12-2a1d-4d02-9b0b-abb058edc160-kube-api-access-2j5dg\") pod \"fa687d12-2a1d-4d02-9b0b-abb058edc160\" (UID: \"fa687d12-2a1d-4d02-9b0b-abb058edc160\") " Oct 31 16:11:40.470035 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.469783 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa687d12-2a1d-4d02-9b0b-abb058edc160-plugins-config-volume" pod "fa687d12-2a1d-4d02-9b0b-abb058edc160" (UID: "fa687d12-2a1d-4d02-9b0b-abb058edc160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:40.471930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.471908 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa687d12-2a1d-4d02-9b0b-abb058edc160-kube-api-access-2j5dg" pod "fa687d12-2a1d-4d02-9b0b-abb058edc160" (UID: "fa687d12-2a1d-4d02-9b0b-abb058edc160"). InnerVolumeSpecName "kube-api-access-2j5dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:40.570645 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.570615 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3008474a-cc15-4493-a59b-363e421502cb-kube-api-access-wtt8n\" (UniqueName: \"kubernetes.io/projected/3008474a-cc15-4493-a59b-363e421502cb-kube-api-access-wtt8n\") pod \"3008474a-cc15-4493-a59b-363e421502cb\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " Oct 31 16:11:40.570846 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.570666 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3008474a-cc15-4493-a59b-363e421502cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3008474a-cc15-4493-a59b-363e421502cb-plugins-config-volume\") pod \"3008474a-cc15-4493-a59b-363e421502cb\" (UID: \"3008474a-cc15-4493-a59b-363e421502cb\") " Oct 31 16:11:40.571049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.571015 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j5dg\" (UniqueName: \"kubernetes.io/projected/fa687d12-2a1d-4d02-9b0b-abb058edc160-kube-api-access-2j5dg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:40.571175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.571146 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3008474a-cc15-4493-a59b-363e421502cb-plugins-config-volume" pod "3008474a-cc15-4493-a59b-363e421502cb" (UID: "3008474a-cc15-4493-a59b-363e421502cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:40.571254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.571164 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa687d12-2a1d-4d02-9b0b-abb058edc160-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:40.573274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.573250 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3008474a-cc15-4493-a59b-363e421502cb-kube-api-access-wtt8n" pod "3008474a-cc15-4493-a59b-363e421502cb" (UID: "3008474a-cc15-4493-a59b-363e421502cb"). InnerVolumeSpecName "kube-api-access-wtt8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:40.672135 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.672045 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtt8n\" (UniqueName: \"kubernetes.io/projected/3008474a-cc15-4493-a59b-363e421502cb-kube-api-access-wtt8n\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:40.672135 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.672074 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3008474a-cc15-4493-a59b-363e421502cb-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:40.811495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:40.809062 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:41.090760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.090718 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:41.326440 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.326399 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" event={"ID":"36531d0f-fb84-402e-a413-a5f1acd95106","Type":"ContainerStarted","Data":"8cb958aaa8b3ff4a781207cbf3a25beed192c2499e56a1d0c7262080731457b0"} Oct 31 16:11:41.327809 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.327783 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" Oct 31 16:11:41.327907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.327826 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j" event={"ID":"3008474a-cc15-4493-a59b-363e421502cb","Type":"ContainerDied","Data":"ef4390a8461211d1c5b67aa8e07a684ab3b01828a3327d80f99dcfd0fa5ce5c4"} Oct 31 16:11:41.327907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.327884 4865 scope.go:122] "RemoveContainer" containerID="396512a9452a9b0ac9834bea1aa6ddbdcb97838dc2b418bae65649681b77855f" Oct 31 16:11:41.329250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.329226 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" event={"ID":"fa687d12-2a1d-4d02-9b0b-abb058edc160","Type":"ContainerDied","Data":"e0487954be3de66d30cbc791d44c8fe0cda01992883c40ce171ad1eb67d75d57"} Oct 31 16:11:41.329250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.329243 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k" Oct 31 16:11:41.337697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.337676 4865 scope.go:122] "RemoveContainer" containerID="4360a034a24f379bf7a0549b7d7755efe1f6adcc065037dde1aa3cb260a18896" Oct 31 16:11:41.383451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.383424 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz"] Oct 31 16:11:41.435806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.435773 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e4bd6cf-vwqfz"] Oct 31 16:11:41.525944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.525871 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" podStartSLOduration=2.525850219 podStartE2EDuration="2.525850219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:41.523768869 +0000 UTC m=+5147.143320284" watchObservedRunningTime="2026-10-31 16:11:41.525850219 +0000 UTC m=+5147.145401631" Oct 31 16:11:41.637351 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.637178 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k"] Oct 31 16:11:41.643716 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.643674 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e4bd6cf-786fccd585-dt98k"] Oct 31 16:11:41.748821 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.748781 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j"] Oct 31 16:11:41.751526 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.751502 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e4bd6cf-8458b5887f-kgn8j"] Oct 31 16:11:41.799411 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.799266 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:41.808362 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.808327 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body= Oct 31 16:11:41.808699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:41.808668 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" Oct 31 16:11:42.496327 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:42.496293 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3008474a-cc15-4493-a59b-363e421502cb" path="/var/lib/kubelet/pods/3008474a-cc15-4493-a59b-363e421502cb/volumes" Oct 31 16:11:42.497296 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:42.497274 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e967ba6d-6b9b-4208-b47b-7de9c1d7e32c" path="/var/lib/kubelet/pods/e967ba6d-6b9b-4208-b47b-7de9c1d7e32c/volumes" Oct 31 16:11:42.500375 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:42.500355 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa687d12-2a1d-4d02-9b0b-abb058edc160" path="/var/lib/kubelet/pods/fa687d12-2a1d-4d02-9b0b-abb058edc160/volumes" Oct 31 16:11:42.818552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:42.817582 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body= Oct 31 16:11:42.818552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:42.817648 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" Oct 31 16:11:43.808924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:43.808883 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body= Oct 31 16:11:43.809260 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:43.809225 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" Oct 31 16:11:44.444190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:44.444154 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:11:44.807348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:44.806902 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body= Oct 31 16:11:44.807348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:44.806968 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" Oct 31 16:11:45.261229 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.260401 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb"] Oct 31 16:11:45.261229 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.260630 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" containerID="cri-o://8cb958aaa8b3ff4a781207cbf3a25beed192c2499e56a1d0c7262080731457b0" gracePeriod=30 Oct 31 16:11:45.267696 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.267665 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:45.283874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.283843 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87"] Oct 31 16:11:45.284069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.284048 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" containerID="cri-o://55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe" gracePeriod=30 Oct 31 16:11:45.319907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.319852 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx"] Oct 31 16:11:45.320440 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.320405 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" containerID="cri-o://a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4" gracePeriod=30 Oct 31 16:11:45.520088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.519994 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:45.962862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.962828 4865 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:11:45.963049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:45.962887 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:11:47.489988 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:47.489945 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:11:50.588151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.588118 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:50.630672 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.630644 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:11:50.702128 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702092 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-podinfo\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702148 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-kube-api-access-tlws4\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-kube-api-access-tlws4\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702184 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-socket\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702222 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-kube-api-access-9fwb9\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-kube-api-access-9fwb9\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702262 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-credential-socket\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702296 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-certs\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702358 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istiod-ca-cert\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702398 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-data\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702430 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-certs\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702483 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-data\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702512 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-envoy\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702549 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-podinfo\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702600 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-socket\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702629 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-credential-socket\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702673 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-token\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-token\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702729 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-token\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-token\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702765 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6339af6-924c-4eac-bb95-cae4777d25a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6339af6-924c-4eac-bb95-cae4777d25a0-istiod-ca-cert\") pod \"d6339af6-924c-4eac-bb95-cae4777d25a0\" (UID: \"d6339af6-924c-4eac-bb95-cae4777d25a0\") " Oct 31 16:11:50.702802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702803 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-envoy\") pod \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\" (UID: \"e263ce0c-7c4f-4480-aefa-c4021fff9d16\") " Oct 31 16:11:50.703488 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.702929 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-data" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.703488 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.703096 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-socket" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.703488 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.703118 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.703488 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.703368 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-certs" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.703739 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.703552 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-data" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.704526 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-certs" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.704860 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6339af6-924c-4eac-bb95-cae4777d25a0-istiod-ca-cert" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.705170 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-credential-socket" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.705631 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istiod-ca-cert" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.705974 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-socket" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.706008 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-envoy" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.706070 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-podinfo" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:11:50.706309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.706280 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-credential-socket" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.707564 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.707516 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-kube-api-access-tlws4" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "kube-api-access-tlws4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:50.707841 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.707815 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-envoy" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.708213 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.708189 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-kube-api-access-9fwb9" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "kube-api-access-9fwb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:50.708937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.708916 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-token" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:50.709054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.709003 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-podinfo" pod "e263ce0c-7c4f-4480-aefa-c4021fff9d16" (UID: "e263ce0c-7c4f-4480-aefa-c4021fff9d16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:11:50.710277 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.710233 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-token" pod "d6339af6-924c-4eac-bb95-cae4777d25a0" (UID: "d6339af6-924c-4eac-bb95-cae4777d25a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:50.804464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804429 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804459 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804490 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6339af6-924c-4eac-bb95-cae4777d25a0-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804502 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804512 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804523 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlws4\" (UniqueName: \"kubernetes.io/projected/d6339af6-924c-4eac-bb95-cae4777d25a0-kube-api-access-tlws4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804534 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804563 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fwb9\" (UniqueName: \"kubernetes.io/projected/e263ce0c-7c4f-4480-aefa-c4021fff9d16-kube-api-access-9fwb9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804574 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804582 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804590 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804598 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804605 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804613 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6339af6-924c-4eac-bb95-cae4777d25a0-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804621 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263ce0c-7c4f-4480-aefa-c4021fff9d16-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804628 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.804760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:50.804638 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263ce0c-7c4f-4480-aefa-c4021fff9d16-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.207551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.207507 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-dff98979c-gmjhh"] Oct 31 16:11:51.208182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208156 4865 cpu_manager.go:414] "containerMap: removing container" podUID="3008474a-cc15-4493-a59b-363e421502cb" containerName="payload-pre-processing" Oct 31 16:11:51.208243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208187 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3008474a-cc15-4493-a59b-363e421502cb" containerName="payload-pre-processing" Oct 31 16:11:51.208243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208207 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" Oct 31 16:11:51.208243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208216 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" Oct 31 16:11:51.208243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208232 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" Oct 31 16:11:51.208243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208240 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" Oct 31 16:11:51.208415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208255 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fa687d12-2a1d-4d02-9b0b-abb058edc160" containerName="payload-processing" Oct 31 16:11:51.208415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208263 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa687d12-2a1d-4d02-9b0b-abb058edc160" containerName="payload-processing" Oct 31 16:11:51.208415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208281 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" Oct 31 16:11:51.208415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208289 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" Oct 31 16:11:51.208415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208386 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="3008474a-cc15-4493-a59b-363e421502cb" containerName="payload-pre-processing" Oct 31 16:11:51.208415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208403 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerName="istio-proxy" Oct 31 16:11:51.208415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208413 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" Oct 31 16:11:51.208673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208422 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa687d12-2a1d-4d02-9b0b-abb058edc160" containerName="payload-processing" Oct 31 16:11:51.208673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.208433 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerName="istio-proxy" Oct 31 16:11:51.213211 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.213191 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.266553 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.266524 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dff98979c-gmjhh"] Oct 31 16:11:51.309840 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.309740 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpj7\" (UniqueName: \"kubernetes.io/projected/05783b11-ae47-4d61-ae54-e0c9c1f505f5-kube-api-access-hnpj7\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.309840 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.309778 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-webhook-cert\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.310074 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.309890 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-metrics-certs\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.312074 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.312054 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 16:11:51.312309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.312285 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 16:11:51.312419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.312326 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b5bsw\"" Oct 31 16:11:51.312419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.312326 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 16:11:51.387461 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.387421 4865 generic.go:356] "Generic (PLEG): container finished" podID="d6339af6-924c-4eac-bb95-cae4777d25a0" containerID="55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe" exitCode=0 Oct 31 16:11:51.387686 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.387499 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" Oct 31 16:11:51.387686 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.387519 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" event={"ID":"d6339af6-924c-4eac-bb95-cae4777d25a0","Type":"ContainerDied","Data":"55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe"} Oct 31 16:11:51.387686 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.387577 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87" event={"ID":"d6339af6-924c-4eac-bb95-cae4777d25a0","Type":"ContainerDied","Data":"3a5608062742adc7240b78c567ca5fb5d4d5e8c7d276bddb09071e9bd720087d"} Oct 31 16:11:51.387686 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.387610 4865 scope.go:122] "RemoveContainer" containerID="55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe" Oct 31 16:11:51.390333 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.390305 4865 generic.go:356] "Generic (PLEG): container finished" podID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" containerID="a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4" exitCode=0 Oct 31 16:11:51.390503 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.390414 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" event={"ID":"e263ce0c-7c4f-4480-aefa-c4021fff9d16","Type":"ContainerDied","Data":"a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4"} Oct 31 16:11:51.390695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.390679 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" Oct 31 16:11:51.391532 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.390463 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx" event={"ID":"e263ce0c-7c4f-4480-aefa-c4021fff9d16","Type":"ContainerDied","Data":"ea6d0627128bb4af6b2c22856ad176e141877c7cb6ec440f81551048450a1135"} Oct 31 16:11:51.393194 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.393172 4865 generic.go:356] "Generic (PLEG): container finished" podID="36531d0f-fb84-402e-a413-a5f1acd95106" containerID="8cb958aaa8b3ff4a781207cbf3a25beed192c2499e56a1d0c7262080731457b0" exitCode=0 Oct 31 16:11:51.393307 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.393238 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" event={"ID":"36531d0f-fb84-402e-a413-a5f1acd95106","Type":"ContainerDied","Data":"8cb958aaa8b3ff4a781207cbf3a25beed192c2499e56a1d0c7262080731457b0"} Oct 31 16:11:51.410716 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.410680 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-metrics-certs\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.411287 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.410833 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpj7\" (UniqueName: \"kubernetes.io/projected/05783b11-ae47-4d61-ae54-e0c9c1f505f5-kube-api-access-hnpj7\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.411287 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.410883 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-webhook-cert\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.413190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.412974 4865 scope.go:122] "RemoveContainer" containerID="55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe" Oct 31 16:11:51.413740 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:51.413719 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe\": container with ID starting with 55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe not found: ID does not exist" containerID="55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe" Oct 31 16:11:51.413858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.413754 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe"} err="failed to get container status \"55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe\": rpc error: code = NotFound desc = could not find container \"55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe\": container with ID starting with 55f8070fa6b9a5dec99cb3c2202b36aedfc5229aef277b4114e062ab5c8a7bfe not found: ID does not exist" Oct 31 16:11:51.413858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.413780 4865 scope.go:122] "RemoveContainer" containerID="a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4" Oct 31 16:11:51.414575 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.414555 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-metrics-certs\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.415392 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.415369 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-webhook-cert\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.446188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.446162 4865 scope.go:122] "RemoveContainer" containerID="8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62" Oct 31 16:11:51.456884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.456857 4865 scope.go:122] "RemoveContainer" containerID="a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4" Oct 31 16:11:51.457230 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:51.457203 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4\": container with ID starting with a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4 not found: ID does not exist" containerID="a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4" Oct 31 16:11:51.457362 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.457243 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4"} err="failed to get container status \"a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4\": rpc error: code = NotFound desc = could not find container \"a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4\": container with ID starting with a964e0bf1bc7a751b355f863a9512b0c4e7e1f4ada8a3378ea3c5100664d0ed4 not found: ID does not exist" Oct 31 16:11:51.457362 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.457269 4865 scope.go:122] "RemoveContainer" containerID="8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62" Oct 31 16:11:51.457791 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:11:51.457737 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62\": container with ID starting with 8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62 not found: ID does not exist" containerID="8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62" Oct 31 16:11:51.457874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.457777 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62"} err="failed to get container status \"8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62\": rpc error: code = NotFound desc = could not find container \"8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62\": container with ID starting with 8fd6a0b361c72748a6b032a608f20f29524875190cdeb5381ba63faa6c481d62 not found: ID does not exist" Oct 31 16:11:51.471401 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.471361 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpj7\" (UniqueName: \"kubernetes.io/projected/05783b11-ae47-4d61-ae54-e0c9c1f505f5-kube-api-access-hnpj7\") pod \"maas-controller-dff98979c-gmjhh\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.508964 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.508937 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:51.526640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.526612 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:51.553138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.553096 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx"] Oct 31 16:11:51.556083 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.556054 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-n7dfx"] Oct 31 16:11:51.612441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612404 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-data\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612450 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-certs\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612506 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-kube-api-access-pm9nv\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-kube-api-access-pm9nv\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612545 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-envoy\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612579 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36531d0f-fb84-402e-a413-a5f1acd95106-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36531d0f-fb84-402e-a413-a5f1acd95106-istio-podinfo\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612644 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-credential-socket\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612670 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36531d0f-fb84-402e-a413-a5f1acd95106-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36531d0f-fb84-402e-a413-a5f1acd95106-istiod-ca-cert\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612724 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-istio-token\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-istio-token\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.612944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.612817 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-socket\") pod \"36531d0f-fb84-402e-a413-a5f1acd95106\" (UID: \"36531d0f-fb84-402e-a413-a5f1acd95106\") " Oct 31 16:11:51.613980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.613629 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-credential-socket" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.613980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.613641 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-certs" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.613980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.613832 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-data" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.614417 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.613998 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-socket" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.614417 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.614079 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36531d0f-fb84-402e-a413-a5f1acd95106-istiod-ca-cert" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:51.615964 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.615901 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-istio-token" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:51.616080 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.615958 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36531d0f-fb84-402e-a413-a5f1acd95106-istio-podinfo" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:11:51.616320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.616295 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-envoy" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.616419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.616360 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-kube-api-access-pm9nv" pod "36531d0f-fb84-402e-a413-a5f1acd95106" (UID: "36531d0f-fb84-402e-a413-a5f1acd95106"). InnerVolumeSpecName "kube-api-access-pm9nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:51.664564 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.664529 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87"] Oct 31 16:11:51.666344 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.666321 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-mtn87"] Oct 31 16:11:51.714255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714164 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714204 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714223 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9nv\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-kube-api-access-pm9nv\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714238 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714260 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36531d0f-fb84-402e-a413-a5f1acd95106-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714277 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714292 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36531d0f-fb84-402e-a413-a5f1acd95106-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714305 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36531d0f-fb84-402e-a413-a5f1acd95106-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.714568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.714319 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36531d0f-fb84-402e-a413-a5f1acd95106-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.898807 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.898782 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dff98979c-gmjhh"] Oct 31 16:11:51.900640 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:11:51.900605 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05783b11_ae47_4d61_ae54_e0c9c1f505f5.slice/crio-45423f965ae572eca9837ddd422a41143405166a9662427aba2a7ffa512686bc WatchSource:0}: Error finding container 45423f965ae572eca9837ddd422a41143405166a9662427aba2a7ffa512686bc: Status 404 returned error can't find the container with id 45423f965ae572eca9837ddd422a41143405166a9662427aba2a7ffa512686bc Oct 31 16:11:51.902444 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:51.902429 4865 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:11:52.402641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.402606 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" Oct 31 16:11:52.402924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.402629 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb" event={"ID":"36531d0f-fb84-402e-a413-a5f1acd95106","Type":"ContainerDied","Data":"2d596ae8409f82eb8abbaba173831addbd91fdb35739f87f5d90b56106a85144"} Oct 31 16:11:52.403066 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.402979 4865 scope.go:122] "RemoveContainer" containerID="8cb958aaa8b3ff4a781207cbf3a25beed192c2499e56a1d0c7262080731457b0" Oct 31 16:11:52.404087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.404053 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-gmjhh" event={"ID":"05783b11-ae47-4d61-ae54-e0c9c1f505f5","Type":"ContainerStarted","Data":"45423f965ae572eca9837ddd422a41143405166a9662427aba2a7ffa512686bc"} Oct 31 16:11:52.495370 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.495329 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6339af6-924c-4eac-bb95-cae4777d25a0" path="/var/lib/kubelet/pods/d6339af6-924c-4eac-bb95-cae4777d25a0/volumes" Oct 31 16:11:52.496024 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.496003 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e263ce0c-7c4f-4480-aefa-c4021fff9d16" path="/var/lib/kubelet/pods/e263ce0c-7c4f-4480-aefa-c4021fff9d16/volumes" Oct 31 16:11:52.566602 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.566563 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb"] Oct 31 16:11:52.569640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:52.569577 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-z55rb"] Oct 31 16:11:53.288692 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:53.288633 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:11:53.289219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:53.288742 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:11:54.495582 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:54.495549 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" path="/var/lib/kubelet/pods/36531d0f-fb84-402e-a413-a5f1acd95106/volumes" Oct 31 16:11:55.119620 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:55.119587 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 16:11:55.431193 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:55.431100 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-gmjhh" event={"ID":"05783b11-ae47-4d61-ae54-e0c9c1f505f5","Type":"ContainerStarted","Data":"ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01"} Oct 31 16:11:55.431388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:55.431227 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:11:55.546708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:11:55.546649 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-dff98979c-gmjhh" podStartSLOduration=1.381240295 podStartE2EDuration="4.546633246s" totalImagesPullingTime="3.165392951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:55.544310071 +0000 UTC m=+5161.163861482" watchObservedRunningTime="2026-10-31 16:11:55.546633246 +0000 UTC m=+5161.166184696" Oct 31 16:12:00.319281 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:00.319240 4865 scope.go:122] "RemoveContainer" containerID="1b59652f18c07fbf803b64fdca74796d0008c5dcf6e4e237875f97ed86cbe754" Oct 31 16:12:00.328737 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:00.328718 4865 scope.go:122] "RemoveContainer" containerID="b2c675b6dde40b6727a4e7672232dd93b92b84ece717761412a6285c26906064" Oct 31 16:12:00.337802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:00.337776 4865 scope.go:122] "RemoveContainer" containerID="9c30c1bf67af6b72b3115276f5b864e095f33e445e1ce2ba9ff8a30f029a042d" Oct 31 16:12:00.345549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:00.345530 4865 scope.go:122] "RemoveContainer" containerID="c56c0a78088b4b1964e2588de0b154c3b2eab412797014ecace1427fa4530d0c" Oct 31 16:12:00.353263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:00.353242 4865 scope.go:122] "RemoveContainer" containerID="a5f98a7e7beeaad2902736b3008166b8d508452087b6197c1300d41c392c67f1" Oct 31 16:12:06.439038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:06.438956 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:12:08.927206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:08.927172 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p"] Oct 31 16:12:08.927657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:08.927576 4865 cpu_manager.go:414] "containerMap: removing container" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" Oct 31 16:12:08.927657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:08.927588 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" Oct 31 16:12:08.927657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:08.927653 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="36531d0f-fb84-402e-a413-a5f1acd95106" containerName="istio-proxy" Oct 31 16:12:08.930715 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:08.930698 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075453 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075521 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075774 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075601 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075774 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075633 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zq6\" (UniqueName: \"kubernetes.io/projected/0d25046d-9187-4e7c-b413-612b4446cb3a-kube-api-access-d9zq6\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075774 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075744 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d25046d-9187-4e7c-b413-612b4446cb3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075934 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075784 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075934 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075802 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.075934 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075915 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.076077 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.075943 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.130580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.130540 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p"] Oct 31 16:12:09.176721 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.176681 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.176910 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.176734 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.176910 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.176865 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.176910 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.176906 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.176930 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zq6\" (UniqueName: \"kubernetes.io/projected/0d25046d-9187-4e7c-b413-612b4446cb3a-kube-api-access-d9zq6\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.176989 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d25046d-9187-4e7c-b413-612b4446cb3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177030 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177056 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177146 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177212 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177519 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177488 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177606 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177579 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177626 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.177952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.177926 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d25046d-9187-4e7c-b413-612b4446cb3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.179397 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.179365 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.180109 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.180090 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.234202 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.234173 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zq6\" (UniqueName: \"kubernetes.io/projected/0d25046d-9187-4e7c-b413-612b4446cb3a-kube-api-access-d9zq6\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.234375 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.234175 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d25046d-9187-4e7c-b413-612b4446cb3a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bgh7p\" (UID: \"0d25046d-9187-4e7c-b413-612b4446cb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.244012 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.243984 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:09.667673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.667643 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p"] Oct 31 16:12:09.668227 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:12:09.668202 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d25046d_9187_4e7c_b413_612b4446cb3a.slice/crio-0ce03c4fc3b31b471c4377dce73ca69b8a41b5e49a4ccc2da19a26c83c6edfe6 WatchSource:0}: Error finding container 0ce03c4fc3b31b471c4377dce73ca69b8a41b5e49a4ccc2da19a26c83c6edfe6: Status 404 returned error can't find the container with id 0ce03c4fc3b31b471c4377dce73ca69b8a41b5e49a4ccc2da19a26c83c6edfe6 Oct 31 16:12:09.670642 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.670609 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:12:09.670745 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.670680 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:12:09.670745 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:09.670711 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:12:10.525885 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:10.525847 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" event={"ID":"0d25046d-9187-4e7c-b413-612b4446cb3a","Type":"ContainerStarted","Data":"e3ce66cdd4293e1c21f918c9da1ea60ae761dcef195c2a455e745d4dafd1137b"} Oct 31 16:12:10.526355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:10.525917 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" event={"ID":"0d25046d-9187-4e7c-b413-612b4446cb3a","Type":"ContainerStarted","Data":"0ce03c4fc3b31b471c4377dce73ca69b8a41b5e49a4ccc2da19a26c83c6edfe6"} Oct 31 16:12:10.690965 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:10.690897 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podStartSLOduration=2.690877941 podStartE2EDuration="2.690877941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:10.689107534 +0000 UTC m=+5176.308658945" watchObservedRunningTime="2026-10-31 16:12:10.690877941 +0000 UTC m=+5176.310429353" Oct 31 16:12:11.245177 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:11.245143 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:11.246719 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:11.246691 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 31 16:12:11.246875 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:11.246739 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 31 16:12:12.245133 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:12.245092 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 31 16:12:12.245618 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:12.245162 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 31 16:12:13.244753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:13.244714 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 31 16:12:13.244946 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:13.244793 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 31 16:12:14.244944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:14.244905 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 31 16:12:14.245389 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:14.244970 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 31 16:12:14.619409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:14.619319 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:15.244635 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:15.244593 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 31 16:12:15.244865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:15.244660 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 31 16:12:16.077464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.077393 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:16.167882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.167840 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9"] Oct 31 16:12:16.172124 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.172103 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.226373 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.226338 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9"] Oct 31 16:12:16.244691 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.244661 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 31 16:12:16.244883 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.244717 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 31 16:12:16.266165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.266120 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.266165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.266161 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbwg\" (UniqueName: \"kubernetes.io/projected/d64a13f5-2983-47e5-bd3b-8e5438885c59-kube-api-access-dmbwg\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.266442 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.266183 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.270829 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.270802 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:12:16.270963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.270882 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:12:16.270963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.270881 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:12:16.270963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.270941 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:12:16.367136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.367052 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.367136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.367097 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbwg\" (UniqueName: \"kubernetes.io/projected/d64a13f5-2983-47e5-bd3b-8e5438885c59-kube-api-access-dmbwg\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.367136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.367134 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.367931 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.367908 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.369702 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.369684 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.423626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.423595 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbwg\" (UniqueName: \"kubernetes.io/projected/d64a13f5-2983-47e5-bd3b-8e5438885c59-kube-api-access-dmbwg\") pod \"maas-api-revoke-sub-85237e45-czwn9\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.484917 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.484879 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:16.860157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:16.860124 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9"] Oct 31 16:12:16.861098 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:12:16.861065 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64a13f5_2983_47e5_bd3b_8e5438885c59.slice/crio-f4b98bd22a576ca738d0a3d6f73113ba6c19b14aca24b5caf0b8aa9519a92a42 WatchSource:0}: Error finding container f4b98bd22a576ca738d0a3d6f73113ba6c19b14aca24b5caf0b8aa9519a92a42: Status 404 returned error can't find the container with id f4b98bd22a576ca738d0a3d6f73113ba6c19b14aca24b5caf0b8aa9519a92a42 Oct 31 16:12:17.245162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:17.245125 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body= Oct 31 16:12:17.245552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:17.245185 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" Oct 31 16:12:17.489950 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:12:17.489917 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:12:17.567129 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:17.567088 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" event={"ID":"d64a13f5-2983-47e5-bd3b-8e5438885c59","Type":"ContainerStarted","Data":"2093ef5733ca5f699989a7d0276f383ce1312af6dfc433b4497cf966133541c5"} Oct 31 16:12:17.567324 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:17.567147 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" event={"ID":"d64a13f5-2983-47e5-bd3b-8e5438885c59","Type":"ContainerStarted","Data":"f4b98bd22a576ca738d0a3d6f73113ba6c19b14aca24b5caf0b8aa9519a92a42"} Oct 31 16:12:17.679801 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:17.679569 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" podStartSLOduration=1.679550876 podStartE2EDuration="1.679550876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:17.678171083 +0000 UTC m=+5183.297722498" watchObservedRunningTime="2026-10-31 16:12:17.679550876 +0000 UTC m=+5183.299102287" Oct 31 16:12:17.691896 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:17.691844 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:18.575010 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:18.574972 4865 generic.go:356] "Generic (PLEG): container finished" podID="d64a13f5-2983-47e5-bd3b-8e5438885c59" containerID="2093ef5733ca5f699989a7d0276f383ce1312af6dfc433b4497cf966133541c5" exitCode=0 Oct 31 16:12:18.575511 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:18.575072 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" event={"ID":"d64a13f5-2983-47e5-bd3b-8e5438885c59","Type":"ContainerDied","Data":"2093ef5733ca5f699989a7d0276f383ce1312af6dfc433b4497cf966133541c5"} Oct 31 16:12:18.982073 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:18.982041 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:18.982610 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:18.982588 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:18.983419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:18.983396 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" Oct 31 16:12:19.708878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.708856 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:19.800565 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.800529 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-openshift-ca\") pod \"d64a13f5-2983-47e5-bd3b-8e5438885c59\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " Oct 31 16:12:19.800783 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.800583 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64a13f5-2983-47e5-bd3b-8e5438885c59-kube-api-access-dmbwg\" (UniqueName: \"kubernetes.io/projected/d64a13f5-2983-47e5-bd3b-8e5438885c59-kube-api-access-dmbwg\") pod \"d64a13f5-2983-47e5-bd3b-8e5438885c59\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " Oct 31 16:12:19.800783 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.800644 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-xks-ca\") pod \"d64a13f5-2983-47e5-bd3b-8e5438885c59\" (UID: \"d64a13f5-2983-47e5-bd3b-8e5438885c59\") " Oct 31 16:12:19.800977 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.800953 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-openshift-ca" pod "d64a13f5-2983-47e5-bd3b-8e5438885c59" (UID: "d64a13f5-2983-47e5-bd3b-8e5438885c59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:12:19.802969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.802937 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64a13f5-2983-47e5-bd3b-8e5438885c59-kube-api-access-dmbwg" pod "d64a13f5-2983-47e5-bd3b-8e5438885c59" (UID: "d64a13f5-2983-47e5-bd3b-8e5438885c59"). InnerVolumeSpecName "kube-api-access-dmbwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:12:19.803072 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.802940 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-xks-ca" pod "d64a13f5-2983-47e5-bd3b-8e5438885c59" (UID: "d64a13f5-2983-47e5-bd3b-8e5438885c59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:12:19.901901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.901816 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:12:19.901901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.901850 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmbwg\" (UniqueName: \"kubernetes.io/projected/d64a13f5-2983-47e5-bd3b-8e5438885c59-kube-api-access-dmbwg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:12:19.901901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:19.901864 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64a13f5-2983-47e5-bd3b-8e5438885c59-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:12:20.586310 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:20.586272 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" Oct 31 16:12:20.586518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:20.586265 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9" event={"ID":"d64a13f5-2983-47e5-bd3b-8e5438885c59","Type":"ContainerDied","Data":"f4b98bd22a576ca738d0a3d6f73113ba6c19b14aca24b5caf0b8aa9519a92a42"} Oct 31 16:12:20.586518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:20.586435 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b98bd22a576ca738d0a3d6f73113ba6c19b14aca24b5caf0b8aa9519a92a42" Oct 31 16:12:28.592673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:28.592629 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:28.718271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:28.718231 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dff98979c-gmjhh"] Oct 31 16:12:28.718496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:28.718437 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-dff98979c-gmjhh" podUID="05783b11-ae47-4d61-ae54-e0c9c1f505f5" containerName="manager" containerID="cri-o://ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01" gracePeriod=30 Oct 31 16:12:28.971768 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:28.971740 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:12:29.081734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.081692 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-webhook-cert\") pod \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " Oct 31 16:12:29.081962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.081774 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05783b11-ae47-4d61-ae54-e0c9c1f505f5-kube-api-access-hnpj7\" (UniqueName: \"kubernetes.io/projected/05783b11-ae47-4d61-ae54-e0c9c1f505f5-kube-api-access-hnpj7\") pod \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " Oct 31 16:12:29.081962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.081806 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-metrics-certs\") pod \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\" (UID: \"05783b11-ae47-4d61-ae54-e0c9c1f505f5\") " Oct 31 16:12:29.084241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.084207 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-webhook-cert" pod "05783b11-ae47-4d61-ae54-e0c9c1f505f5" (UID: "05783b11-ae47-4d61-ae54-e0c9c1f505f5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:12:29.084397 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.084252 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-metrics-certs" pod "05783b11-ae47-4d61-ae54-e0c9c1f505f5" (UID: "05783b11-ae47-4d61-ae54-e0c9c1f505f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:12:29.084397 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.084321 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05783b11-ae47-4d61-ae54-e0c9c1f505f5-kube-api-access-hnpj7" pod "05783b11-ae47-4d61-ae54-e0c9c1f505f5" (UID: "05783b11-ae47-4d61-ae54-e0c9c1f505f5"). InnerVolumeSpecName "kube-api-access-hnpj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:12:29.183188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.183101 4865 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-webhook-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:12:29.183188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.183138 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpj7\" (UniqueName: \"kubernetes.io/projected/05783b11-ae47-4d61-ae54-e0c9c1f505f5-kube-api-access-hnpj7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:12:29.183188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.183153 4865 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05783b11-ae47-4d61-ae54-e0c9c1f505f5-metrics-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:12:29.640590 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.640555 4865 generic.go:356] "Generic (PLEG): container finished" podID="05783b11-ae47-4d61-ae54-e0c9c1f505f5" containerID="ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01" exitCode=0 Oct 31 16:12:29.641111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.640605 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-gmjhh" Oct 31 16:12:29.641111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.640646 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-gmjhh" event={"ID":"05783b11-ae47-4d61-ae54-e0c9c1f505f5","Type":"ContainerDied","Data":"ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01"} Oct 31 16:12:29.641111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.640710 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-gmjhh" event={"ID":"05783b11-ae47-4d61-ae54-e0c9c1f505f5","Type":"ContainerDied","Data":"45423f965ae572eca9837ddd422a41143405166a9662427aba2a7ffa512686bc"} Oct 31 16:12:29.641111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.640741 4865 scope.go:122] "RemoveContainer" containerID="ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01" Oct 31 16:12:29.650812 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.650793 4865 scope.go:122] "RemoveContainer" containerID="ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01" Oct 31 16:12:29.651108 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:12:29.651090 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01\": container with ID starting with ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01 not found: ID does not exist" containerID="ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01" Oct 31 16:12:29.651165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.651118 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01"} err="failed to get container status \"ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01\": rpc error: code = NotFound desc = could not find container \"ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01\": container with ID starting with ce623ac2545bb90bf6012d5a89dd2ea442ad7516589f5a13423fb31bc6cb9b01 not found: ID does not exist" Oct 31 16:12:29.803876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.803837 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dff98979c-gmjhh"] Oct 31 16:12:29.853357 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:29.853324 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-dff98979c-gmjhh"] Oct 31 16:12:30.494617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:30.494587 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05783b11-ae47-4d61-ae54-e0c9c1f505f5" path="/var/lib/kubelet/pods/05783b11-ae47-4d61-ae54-e0c9c1f505f5/volumes" Oct 31 16:12:31.020257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.020218 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:31.186564 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.186531 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-dff98979c-mrlql"] Oct 31 16:12:31.186916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.186902 4865 cpu_manager.go:414] "containerMap: removing container" podUID="05783b11-ae47-4d61-ae54-e0c9c1f505f5" containerName="manager" Oct 31 16:12:31.186916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.186915 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05783b11-ae47-4d61-ae54-e0c9c1f505f5" containerName="manager" Oct 31 16:12:31.187020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.186938 4865 cpu_manager.go:414] "containerMap: removing container" podUID="d64a13f5-2983-47e5-bd3b-8e5438885c59" containerName="revoke-keys" Oct 31 16:12:31.187020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.186944 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64a13f5-2983-47e5-bd3b-8e5438885c59" containerName="revoke-keys" Oct 31 16:12:31.187020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.187007 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64a13f5-2983-47e5-bd3b-8e5438885c59" containerName="revoke-keys" Oct 31 16:12:31.187020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.187019 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="05783b11-ae47-4d61-ae54-e0c9c1f505f5" containerName="manager" Oct 31 16:12:31.193619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.193597 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.249938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.249904 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b5bsw\"" Oct 31 16:12:31.298703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.298614 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 16:12:31.298703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.298663 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 16:12:31.298902 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.298712 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 16:12:31.302684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.302663 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dff98979c-mrlql"] Oct 31 16:12:31.306117 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.306094 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c1b19d-24d1-429b-ab1a-955010b8c9d4-webhook-cert\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.306249 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.306226 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssf4\" (UniqueName: \"kubernetes.io/projected/83c1b19d-24d1-429b-ab1a-955010b8c9d4-kube-api-access-bssf4\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.306298 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.306266 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c1b19d-24d1-429b-ab1a-955010b8c9d4-metrics-certs\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.406979 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.406941 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssf4\" (UniqueName: \"kubernetes.io/projected/83c1b19d-24d1-429b-ab1a-955010b8c9d4-kube-api-access-bssf4\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.406979 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.406983 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c1b19d-24d1-429b-ab1a-955010b8c9d4-metrics-certs\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.407268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.407017 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c1b19d-24d1-429b-ab1a-955010b8c9d4-webhook-cert\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.409763 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.409738 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c1b19d-24d1-429b-ab1a-955010b8c9d4-metrics-certs\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.409879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.409803 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c1b19d-24d1-429b-ab1a-955010b8c9d4-webhook-cert\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.463928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.463895 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssf4\" (UniqueName: \"kubernetes.io/projected/83c1b19d-24d1-429b-ab1a-955010b8c9d4-kube-api-access-bssf4\") pod \"maas-controller-dff98979c-mrlql\" (UID: \"83c1b19d-24d1-429b-ab1a-955010b8c9d4\") " pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.506975 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.506942 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:31.877523 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:31.877497 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dff98979c-mrlql"] Oct 31 16:12:31.878685 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:12:31.878661 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c1b19d_24d1_429b_ab1a_955010b8c9d4.slice/crio-66facdbf9146b4927074860940b543e2909e1da87249f94c82350f6ea350041a WatchSource:0}: Error finding container 66facdbf9146b4927074860940b543e2909e1da87249f94c82350f6ea350041a: Status 404 returned error can't find the container with id 66facdbf9146b4927074860940b543e2909e1da87249f94c82350f6ea350041a Oct 31 16:12:32.657649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:32.657613 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-mrlql" event={"ID":"83c1b19d-24d1-429b-ab1a-955010b8c9d4","Type":"ContainerStarted","Data":"66facdbf9146b4927074860940b543e2909e1da87249f94c82350f6ea350041a"} Oct 31 16:12:33.096821 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:33.096787 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:33.214549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:33.214514 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:33.666430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:33.666389 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-mrlql" event={"ID":"83c1b19d-24d1-429b-ab1a-955010b8c9d4","Type":"ContainerStarted","Data":"050d63d2189335ba2176cc7601ba9b92f83e2ab611c279b43768f40e2253d568"} Oct 31 16:12:33.666900 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:33.666490 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:33.810449 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:33.810390 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-dff98979c-mrlql" podStartSLOduration=2.048982459 podStartE2EDuration="2.810374074s" totalImagesPullingTime="761.391615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:33.80679029 +0000 UTC m=+5199.426341691" watchObservedRunningTime="2026-10-31 16:12:33.810374074 +0000 UTC m=+5199.429925484" Oct 31 16:12:39.046233 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:39.046199 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r"] Oct 31 16:12:39.096286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:39.096253 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-736a0494-wg79r"] Oct 31 16:12:40.494609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:40.494572 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99980790-5801-4522-822b-27a52269fff9" path="/var/lib/kubelet/pods/99980790-5801-4522-822b-27a52269fff9/volumes" Oct 31 16:12:44.673984 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:44.673951 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-dff98979c-mrlql" Oct 31 16:12:47.489940 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:12:47.489900 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:12:51.045199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:51.045163 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5"] Oct 31 16:12:51.098617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:51.098567 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767911c0-skrk5"] Oct 31 16:12:52.495586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:52.495552 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e864631b-8ce8-4890-90dd-4965656c9333" path="/var/lib/kubelet/pods/e864631b-8ce8-4890-90dd-4965656c9333/volumes" Oct 31 16:12:56.618916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:12:56.618878 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:13:00.404285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:00.404250 4865 scope.go:122] "RemoveContainer" containerID="224525d0fec23233b3dfe7bb495b629c02019e3193b2b0b55f5b64ecf142c25e" Oct 31 16:13:00.412986 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:00.412963 4865 scope.go:122] "RemoveContainer" containerID="bb8fb11e92d6a493b9e37643315d919c8f751658c580b4309322e9fa4cc30dc4" Oct 31 16:13:02.044588 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:02.044558 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc"] Oct 31 16:13:02.094278 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:02.094233 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aa6d920-svwnc"] Oct 31 16:13:02.494698 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:02.494667 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2245fb-0f17-4c1c-92ad-fa23be984423" path="/var/lib/kubelet/pods/ae2245fb-0f17-4c1c-92ad-fa23be984423/volumes" Oct 31 16:13:03.518581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.518545 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v"] Oct 31 16:13:03.523696 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.523679 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.578283 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.578234 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v"] Oct 31 16:13:03.623215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.623189 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:13:03.623215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.623194 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:13:03.625188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.625163 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:13:03.625324 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.625266 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:13:03.710429 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.710387 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8mk\" (UniqueName: \"kubernetes.io/projected/e251abfb-45b8-472c-888f-a76561530b4b-kube-api-access-9w8mk\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.710660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.710525 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.710660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.710622 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.812156 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.812050 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8mk\" (UniqueName: \"kubernetes.io/projected/e251abfb-45b8-472c-888f-a76561530b4b-kube-api-access-9w8mk\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.812156 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.812108 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.812436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.812175 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.812854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.812828 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.815134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.815101 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:03.868938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:03.868909 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8mk\" (UniqueName: \"kubernetes.io/projected/e251abfb-45b8-472c-888f-a76561530b4b-kube-api-access-9w8mk\") pod \"maas-api-revoke-sub-f904bff9-mn75v\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:04.136730 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:04.136650 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:04.508052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:04.508022 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v"] Oct 31 16:13:04.510041 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:13:04.510013 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode251abfb_45b8_472c_888f_a76561530b4b.slice/crio-9c0508a928095636b52747af9d9e7d14f9e166b3428887d2c7f0c4ef3b33629d WatchSource:0}: Error finding container 9c0508a928095636b52747af9d9e7d14f9e166b3428887d2c7f0c4ef3b33629d: Status 404 returned error can't find the container with id 9c0508a928095636b52747af9d9e7d14f9e166b3428887d2c7f0c4ef3b33629d Oct 31 16:13:04.826328 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:04.826294 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" event={"ID":"e251abfb-45b8-472c-888f-a76561530b4b","Type":"ContainerStarted","Data":"62c9affe2e8b3c3fe159851c0807e4e0df0ff770b5aca1c505508ebb040b0129"} Oct 31 16:13:04.826687 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:04.826354 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" event={"ID":"e251abfb-45b8-472c-888f-a76561530b4b","Type":"ContainerStarted","Data":"9c0508a928095636b52747af9d9e7d14f9e166b3428887d2c7f0c4ef3b33629d"} Oct 31 16:13:04.939608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:04.939553 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" podStartSLOduration=1.939538805 podStartE2EDuration="1.939538805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:04.938041304 +0000 UTC m=+5230.557592716" watchObservedRunningTime="2026-10-31 16:13:04.939538805 +0000 UTC m=+5230.559090215" Oct 31 16:13:05.833494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:05.833438 4865 generic.go:356] "Generic (PLEG): container finished" podID="e251abfb-45b8-472c-888f-a76561530b4b" containerID="62c9affe2e8b3c3fe159851c0807e4e0df0ff770b5aca1c505508ebb040b0129" exitCode=0 Oct 31 16:13:05.833984 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:05.833521 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" event={"ID":"e251abfb-45b8-472c-888f-a76561530b4b","Type":"ContainerDied","Data":"62c9affe2e8b3c3fe159851c0807e4e0df0ff770b5aca1c505508ebb040b0129"} Oct 31 16:13:06.968884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:06.968852 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:07.041479 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.041447 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-openshift-ca\") pod \"e251abfb-45b8-472c-888f-a76561530b4b\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " Oct 31 16:13:07.041662 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.041516 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e251abfb-45b8-472c-888f-a76561530b4b-kube-api-access-9w8mk\" (UniqueName: \"kubernetes.io/projected/e251abfb-45b8-472c-888f-a76561530b4b-kube-api-access-9w8mk\") pod \"e251abfb-45b8-472c-888f-a76561530b4b\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " Oct 31 16:13:07.041662 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.041597 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-xks-ca\") pod \"e251abfb-45b8-472c-888f-a76561530b4b\" (UID: \"e251abfb-45b8-472c-888f-a76561530b4b\") " Oct 31 16:13:07.041887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.041853 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-openshift-ca" pod "e251abfb-45b8-472c-888f-a76561530b4b" (UID: "e251abfb-45b8-472c-888f-a76561530b4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:13:07.043789 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.043764 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-xks-ca" pod "e251abfb-45b8-472c-888f-a76561530b4b" (UID: "e251abfb-45b8-472c-888f-a76561530b4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:13:07.043891 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.043865 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e251abfb-45b8-472c-888f-a76561530b4b-kube-api-access-9w8mk" pod "e251abfb-45b8-472c-888f-a76561530b4b" (UID: "e251abfb-45b8-472c-888f-a76561530b4b"). InnerVolumeSpecName "kube-api-access-9w8mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:13:07.142944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.142853 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:13:07.142944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.142892 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w8mk\" (UniqueName: \"kubernetes.io/projected/e251abfb-45b8-472c-888f-a76561530b4b-kube-api-access-9w8mk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:13:07.142944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.142903 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e251abfb-45b8-472c-888f-a76561530b4b-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:13:07.845376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.845337 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" Oct 31 16:13:07.845595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.845371 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v" event={"ID":"e251abfb-45b8-472c-888f-a76561530b4b","Type":"ContainerDied","Data":"9c0508a928095636b52747af9d9e7d14f9e166b3428887d2c7f0c4ef3b33629d"} Oct 31 16:13:07.845595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:07.845423 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c0508a928095636b52747af9d9e7d14f9e166b3428887d2c7f0c4ef3b33629d" Oct 31 16:13:09.418664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.418622 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2"] Oct 31 16:13:09.419105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.419002 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e251abfb-45b8-472c-888f-a76561530b4b" containerName="revoke-keys" Oct 31 16:13:09.419105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.419012 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e251abfb-45b8-472c-888f-a76561530b4b" containerName="revoke-keys" Oct 31 16:13:09.419105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.419084 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e251abfb-45b8-472c-888f-a76561530b4b" containerName="revoke-keys" Oct 31 16:13:09.422051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.422034 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.461987 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462022 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwss\" (UniqueName: \"kubernetes.io/projected/1b390bce-baa8-4fdd-895f-c660aafd1813-kube-api-access-bzwss\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462051 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462139 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462234 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462163 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462247 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462299 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462348 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.462525 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.462377 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b390bce-baa8-4fdd-895f-c660aafd1813-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563290 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563253 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563298 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b390bce-baa8-4fdd-895f-c660aafd1813-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563499 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwss\" (UniqueName: \"kubernetes.io/projected/1b390bce-baa8-4fdd-895f-c660aafd1813-kube-api-access-bzwss\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563758 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563556 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563758 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563615 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563758 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563645 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563769 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563773 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563826 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.563935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563854 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.564157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.563933 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.564407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.564379 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b390bce-baa8-4fdd-895f-c660aafd1813-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.564581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.564503 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.567254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.567223 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.573424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.573399 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.625679 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.625612 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b390bce-baa8-4fdd-895f-c660aafd1813-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.625876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.625813 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2"] Oct 31 16:13:09.635343 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.635314 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwss\" (UniqueName: \"kubernetes.io/projected/1b390bce-baa8-4fdd-895f-c660aafd1813-kube-api-access-bzwss\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwld2\" (UID: \"1b390bce-baa8-4fdd-895f-c660aafd1813\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:09.735164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:09.735129 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:10.160971 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:10.160939 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2"] Oct 31 16:13:10.162179 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:13:10.162155 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b390bce_baa8_4fdd_895f_c660aafd1813.slice/crio-88cb2e71de77878b1a35962369f35b4eb806723cbcc2f571791f32fe1a7280ae WatchSource:0}: Error finding container 88cb2e71de77878b1a35962369f35b4eb806723cbcc2f571791f32fe1a7280ae: Status 404 returned error can't find the container with id 88cb2e71de77878b1a35962369f35b4eb806723cbcc2f571791f32fe1a7280ae Oct 31 16:13:10.164263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:10.164229 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:13:10.164363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:10.164296 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:13:10.164363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:10.164327 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:13:10.862783 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:10.862742 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" event={"ID":"1b390bce-baa8-4fdd-895f-c660aafd1813","Type":"ContainerStarted","Data":"2cb75efcac4b9c9dca6163db5f5c7b757562271a42281e4caf796db539e699f5"} Oct 31 16:13:10.863302 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:10.862797 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" event={"ID":"1b390bce-baa8-4fdd-895f-c660aafd1813","Type":"ContainerStarted","Data":"88cb2e71de77878b1a35962369f35b4eb806723cbcc2f571791f32fe1a7280ae"} Oct 31 16:13:11.082210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:11.082144 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podStartSLOduration=2.082130294 podStartE2EDuration="2.082130294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:11.031036611 +0000 UTC m=+5236.650588022" watchObservedRunningTime="2026-10-31 16:13:11.082130294 +0000 UTC m=+5236.701681705" Oct 31 16:13:11.735337 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:11.735300 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:11.737106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:11.737076 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" start-of-body= Oct 31 16:13:11.737254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:11.737127 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" Oct 31 16:13:12.735619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:12.735583 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" start-of-body= Oct 31 16:13:12.736063 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:12.735653 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" Oct 31 16:13:13.049912 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:13.049824 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8"] Oct 31 16:13:13.118888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:13.118847 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-295c6b81-xlwb8"] Oct 31 16:13:13.735815 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:13.735774 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" start-of-body= Oct 31 16:13:13.736270 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:13.735843 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" Oct 31 16:13:14.518862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:14.518825 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84eb97d5-3053-4003-98b2-7abaeee1fb64" path="/var/lib/kubelet/pods/84eb97d5-3053-4003-98b2-7abaeee1fb64/volumes" Oct 31 16:13:14.735719 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:14.735682 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" start-of-body= Oct 31 16:13:14.735907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:14.735739 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" Oct 31 16:13:14.922252 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:14.922135 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:15.735960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:15.735920 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" start-of-body= Oct 31 16:13:15.736428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:15.735976 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" Oct 31 16:13:16.422781 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:16.422712 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:16.531366 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:16.531325 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:13:16.736457 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:16.736419 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" start-of-body= Oct 31 16:13:16.736903 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:16.736503 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" Oct 31 16:13:17.489964 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:13:17.489919 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:13:17.735813 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:17.735778 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" start-of-body= Oct 31 16:13:17.736005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:17.735831 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": dial tcp 10.132.1.152:15021: connect: connection refused" Oct 31 16:13:18.044031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:18.043999 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:19.736986 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:19.736944 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:19.737441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:19.737017 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:20.778635 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:20.778599 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:20.779079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:20.778674 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:21.821689 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:21.821646 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:21.822346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:21.822314 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:22.823355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:22.823311 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:22.824071 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:22.824018 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:23.870404 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:23.870359 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:23.871198 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:23.871158 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:24.348495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:24.347770 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:13:24.741544 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:24.740592 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:24.921649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:24.921602 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:24.922341 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:24.922304 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:25.601911 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:25.601876 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:25.758720 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:25.758689 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" Oct 31 16:13:35.027036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:35.026594 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:35.027036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:35.026666 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:35.832049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:35.832015 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:13:37.642571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.642526 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4"] Oct 31 16:13:37.647917 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.647896 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.701198 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.701167 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4"] Oct 31 16:13:37.715004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.714978 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.715160 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.715051 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgltr\" (UniqueName: \"kubernetes.io/projected/49ffc269-5dcd-41bb-8990-83e32b188b3d-kube-api-access-zgltr\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.715220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.715158 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.746674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.746640 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:13:37.746674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.746650 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:13:37.746925 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.746704 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:13:37.748971 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.748954 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:13:37.815735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.815699 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgltr\" (UniqueName: \"kubernetes.io/projected/49ffc269-5dcd-41bb-8990-83e32b188b3d-kube-api-access-zgltr\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.815930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.815741 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.815930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.815810 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.816539 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.816509 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.818315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.818286 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.872199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.872162 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgltr\" (UniqueName: \"kubernetes.io/projected/49ffc269-5dcd-41bb-8990-83e32b188b3d-kube-api-access-zgltr\") pod \"maas-api-revoke-sub-f03629db-v8pn4\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:37.961460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:37.961430 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:38.332308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:38.332279 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4"] Oct 31 16:13:38.333824 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:13:38.333792 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ffc269_5dcd_41bb_8990_83e32b188b3d.slice/crio-c5f3663681d4ded796c50e4d85c1f3bf55c9c1d464ddd60e75907b2084cd6e78 WatchSource:0}: Error finding container c5f3663681d4ded796c50e4d85c1f3bf55c9c1d464ddd60e75907b2084cd6e78: Status 404 returned error can't find the container with id c5f3663681d4ded796c50e4d85c1f3bf55c9c1d464ddd60e75907b2084cd6e78 Oct 31 16:13:39.088893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.088857 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" event={"ID":"49ffc269-5dcd-41bb-8990-83e32b188b3d","Type":"ContainerStarted","Data":"22aec91943f8b1fe1f29191a50365bb8492a7801f146c98186cc37fcf02542b8"} Oct 31 16:13:39.089338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.088908 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" event={"ID":"49ffc269-5dcd-41bb-8990-83e32b188b3d","Type":"ContainerStarted","Data":"c5f3663681d4ded796c50e4d85c1f3bf55c9c1d464ddd60e75907b2084cd6e78"} Oct 31 16:13:39.201637 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.201587 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" podStartSLOduration=2.201571757 podStartE2EDuration="2.201571757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:39.199966642 +0000 UTC m=+5264.819518052" watchObservedRunningTime="2026-10-31 16:13:39.201571757 +0000 UTC m=+5264.821123213" Oct 31 16:13:39.642289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.642251 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw"] Oct 31 16:13:39.645744 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.645724 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.731882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.731849 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.731893 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed4866e-493a-4027-9fa6-5d4858002f16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.731987 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.732044 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.732077 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6d65\" (UniqueName: \"kubernetes.io/projected/7ed4866e-493a-4027-9fa6-5d4858002f16-kube-api-access-v6d65\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.732099 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.732132 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.732225 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.732369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.732248 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.751569 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.751541 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw"] Oct 31 16:13:39.833088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833051 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833091 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833220 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833264 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed4866e-493a-4027-9fa6-5d4858002f16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833307 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833530 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833336 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833658 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833379 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d65\" (UniqueName: \"kubernetes.io/projected/7ed4866e-493a-4027-9fa6-5d4858002f16-kube-api-access-v6d65\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833696 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833705 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833606 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833902 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833756 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.833979 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.833959 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.834052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.834036 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.834360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.834335 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed4866e-493a-4027-9fa6-5d4858002f16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.835567 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.835549 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.836079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.836053 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.889509 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.889462 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed4866e-493a-4027-9fa6-5d4858002f16-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.889693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.889645 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6d65\" (UniqueName: \"kubernetes.io/projected/7ed4866e-493a-4027-9fa6-5d4858002f16-kube-api-access-v6d65\") pod \"maas-default-gateway-openshift-default-67bf986f99-l6znw\" (UID: \"7ed4866e-493a-4027-9fa6-5d4858002f16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:39.959636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:39.959606 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:40.094902 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:40.094867 4865 generic.go:356] "Generic (PLEG): container finished" podID="49ffc269-5dcd-41bb-8990-83e32b188b3d" containerID="22aec91943f8b1fe1f29191a50365bb8492a7801f146c98186cc37fcf02542b8" exitCode=0 Oct 31 16:13:40.095330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:40.094926 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" event={"ID":"49ffc269-5dcd-41bb-8990-83e32b188b3d","Type":"ContainerDied","Data":"22aec91943f8b1fe1f29191a50365bb8492a7801f146c98186cc37fcf02542b8"} Oct 31 16:13:40.385577 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:40.385547 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw"] Oct 31 16:13:40.386508 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:13:40.386452 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed4866e_493a_4027_9fa6_5d4858002f16.slice/crio-ff3e5a15bc449dd661a56f4efa7643847361729f2e1bf2c77625021d1b1df9c5 WatchSource:0}: Error finding container ff3e5a15bc449dd661a56f4efa7643847361729f2e1bf2c77625021d1b1df9c5: Status 404 returned error can't find the container with id ff3e5a15bc449dd661a56f4efa7643847361729f2e1bf2c77625021d1b1df9c5 Oct 31 16:13:40.388669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:40.388636 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:13:40.388743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:40.388703 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:13:40.388743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:40.388732 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:13:41.122426 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.122377 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" event={"ID":"7ed4866e-493a-4027-9fa6-5d4858002f16","Type":"ContainerStarted","Data":"979d78edd706ac1a4db64d92e69afeaf16d4ab27fd29e3be844539b870de398a"} Oct 31 16:13:41.122927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.122451 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" event={"ID":"7ed4866e-493a-4027-9fa6-5d4858002f16","Type":"ContainerStarted","Data":"ff3e5a15bc449dd661a56f4efa7643847361729f2e1bf2c77625021d1b1df9c5"} Oct 31 16:13:41.277496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.277457 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:41.348848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.348811 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-openshift-ca\") pod \"49ffc269-5dcd-41bb-8990-83e32b188b3d\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " Oct 31 16:13:41.349082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.348856 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ffc269-5dcd-41bb-8990-83e32b188b3d-kube-api-access-zgltr\" (UniqueName: \"kubernetes.io/projected/49ffc269-5dcd-41bb-8990-83e32b188b3d-kube-api-access-zgltr\") pod \"49ffc269-5dcd-41bb-8990-83e32b188b3d\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " Oct 31 16:13:41.349082 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.348901 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-xks-ca\") pod \"49ffc269-5dcd-41bb-8990-83e32b188b3d\" (UID: \"49ffc269-5dcd-41bb-8990-83e32b188b3d\") " Oct 31 16:13:41.349222 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.349174 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-openshift-ca" pod "49ffc269-5dcd-41bb-8990-83e32b188b3d" (UID: "49ffc269-5dcd-41bb-8990-83e32b188b3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:13:41.349370 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.349346 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:13:41.351242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.351216 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-xks-ca" pod "49ffc269-5dcd-41bb-8990-83e32b188b3d" (UID: "49ffc269-5dcd-41bb-8990-83e32b188b3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:13:41.351357 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.351268 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ffc269-5dcd-41bb-8990-83e32b188b3d-kube-api-access-zgltr" pod "49ffc269-5dcd-41bb-8990-83e32b188b3d" (UID: "49ffc269-5dcd-41bb-8990-83e32b188b3d"). InnerVolumeSpecName "kube-api-access-zgltr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:13:41.450209 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.450122 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgltr\" (UniqueName: \"kubernetes.io/projected/49ffc269-5dcd-41bb-8990-83e32b188b3d-kube-api-access-zgltr\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:13:41.450209 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.450168 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ffc269-5dcd-41bb-8990-83e32b188b3d-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:13:41.451599 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.451552 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podStartSLOduration=2.451536557 podStartE2EDuration="2.451536557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:41.335726328 +0000 UTC m=+5266.955277750" watchObservedRunningTime="2026-10-31 16:13:41.451536557 +0000 UTC m=+5267.071087967" Oct 31 16:13:41.960699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.960664 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:41.962623 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.962595 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Oct 31 16:13:41.962785 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:41.962644 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Oct 31 16:13:42.128524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:42.128493 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" Oct 31 16:13:42.128524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:42.128505 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4" event={"ID":"49ffc269-5dcd-41bb-8990-83e32b188b3d","Type":"ContainerDied","Data":"c5f3663681d4ded796c50e4d85c1f3bf55c9c1d464ddd60e75907b2084cd6e78"} Oct 31 16:13:42.129128 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:42.128560 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5f3663681d4ded796c50e4d85c1f3bf55c9c1d464ddd60e75907b2084cd6e78" Oct 31 16:13:42.960707 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:42.960669 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Oct 31 16:13:42.960936 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:42.960731 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Oct 31 16:13:43.960780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:43.960743 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Oct 31 16:13:43.961224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:43.960801 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Oct 31 16:13:44.219798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:44.219686 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:13:44.960545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:44.960510 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Oct 31 16:13:44.960747 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:44.960571 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Oct 31 16:13:45.960676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:45.960640 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Oct 31 16:13:45.961102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:45.960699 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Oct 31 16:13:46.960748 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:46.960712 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Oct 31 16:13:46.961338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:46.960769 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Oct 31 16:13:47.496064 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:13:47.496018 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:13:47.960392 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:47.960354 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Oct 31 16:13:47.960589 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:47.960410 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Oct 31 16:13:49.961124 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:49.961068 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:49.961731 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:49.961126 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:50.962203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:50.962156 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:50.962702 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:50.962258 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:52.003691 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:52.003640 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:52.004280 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:52.003716 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:52.615596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:52.615548 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:52.616095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:52.616070 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:52.616885 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:52.616864 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" Oct 31 16:13:54.160186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:54.160142 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:54.281713 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:54.281673 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:56.654619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:56.654576 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:56.655220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:56.655190 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:57.133162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:57.133122 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:57.363621 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:57.363587 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:57.546448 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:57.546414 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:57.864714 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:57.864624 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:58.006620 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.006586 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:58.088778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.088743 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7"] Oct 31 16:13:58.089287 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.089268 4865 cpu_manager.go:414] "containerMap: removing container" podUID="49ffc269-5dcd-41bb-8990-83e32b188b3d" containerName="revoke-keys" Oct 31 16:13:58.089380 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.089289 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ffc269-5dcd-41bb-8990-83e32b188b3d" containerName="revoke-keys" Oct 31 16:13:58.089380 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.089370 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ffc269-5dcd-41bb-8990-83e32b188b3d" containerName="revoke-keys" Oct 31 16:13:58.093640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.093621 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.149674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.149597 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7"] Oct 31 16:13:58.179811 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.179769 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.180027 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.179960 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzxc\" (UniqueName: \"kubernetes.io/projected/481e2d44-71e6-4242-ac12-5d1193e3e097-kube-api-access-9dzxc\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.180027 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.179994 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.194646 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.194616 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:13:58.194786 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.194658 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:13:58.194786 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.194724 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:13:58.197370 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.197355 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:13:58.280830 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.280794 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.281070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.280906 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzxc\" (UniqueName: \"kubernetes.io/projected/481e2d44-71e6-4242-ac12-5d1193e3e097-kube-api-access-9dzxc\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.281070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.280939 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.281716 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.281688 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.283692 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.283666 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.349037 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.349002 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzxc\" (UniqueName: \"kubernetes.io/projected/481e2d44-71e6-4242-ac12-5d1193e3e097-kube-api-access-9dzxc\") pod \"maas-api-revoke-sub-0165d7d2-m4lx7\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.410913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.410815 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:13:58.789096 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:58.789068 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7"] Oct 31 16:13:58.790228 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:13:58.790191 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481e2d44_71e6_4242_ac12_5d1193e3e097.slice/crio-c7db366a356938e41dda2703b14edb7f393faf3e11ee361659b3ffa1b1cc7113 WatchSource:0}: Error finding container c7db366a356938e41dda2703b14edb7f393faf3e11ee361659b3ffa1b1cc7113: Status 404 returned error can't find the container with id c7db366a356938e41dda2703b14edb7f393faf3e11ee361659b3ffa1b1cc7113 Oct 31 16:13:59.285971 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:59.285931 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:59.360303 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:59.360203 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" event={"ID":"481e2d44-71e6-4242-ac12-5d1193e3e097","Type":"ContainerStarted","Data":"c110b8356044a37c6653a3d307593c978da8f3b69e0519ecd68db795e232f649"} Oct 31 16:13:59.360303 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:59.360264 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" event={"ID":"481e2d44-71e6-4242-ac12-5d1193e3e097","Type":"ContainerStarted","Data":"c7db366a356938e41dda2703b14edb7f393faf3e11ee361659b3ffa1b1cc7113"} Oct 31 16:13:59.479783 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:13:59.479728 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" podStartSLOduration=1.4797134760000001 podStartE2EDuration="1.479713476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:59.476404078 +0000 UTC m=+5285.095955490" watchObservedRunningTime="2026-10-31 16:13:59.479713476 +0000 UTC m=+5285.099264883" Oct 31 16:14:00.367055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:00.367014 4865 generic.go:356] "Generic (PLEG): container finished" podID="481e2d44-71e6-4242-ac12-5d1193e3e097" containerID="c110b8356044a37c6653a3d307593c978da8f3b69e0519ecd68db795e232f649" exitCode=0 Oct 31 16:14:00.367502 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:00.367107 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" event={"ID":"481e2d44-71e6-4242-ac12-5d1193e3e097","Type":"ContainerDied","Data":"c110b8356044a37c6653a3d307593c978da8f3b69e0519ecd68db795e232f649"} Oct 31 16:14:00.440684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:00.440649 4865 scope.go:122] "RemoveContainer" containerID="1316ae5a3551e9ff55cb19a4f6b4b516636c9f975af61fcce839b90bc6a3ef52" Oct 31 16:14:00.449822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:00.449804 4865 scope.go:122] "RemoveContainer" containerID="c1c671fa3e8c82517b73bd0fc7057da675580cf4ca2fde6124d0d7b799288276" Oct 31 16:14:00.458340 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:00.458321 4865 scope.go:122] "RemoveContainer" containerID="de9cb6dc5718f7b9f5441cfeb324e5eed75e9cdef2137e17d6024264174101db" Oct 31 16:14:01.036453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.036421 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:01.173569 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.173524 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:01.510340 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.510316 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:14:01.614657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.614618 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481e2d44-71e6-4242-ac12-5d1193e3e097-kube-api-access-9dzxc\" (UniqueName: \"kubernetes.io/projected/481e2d44-71e6-4242-ac12-5d1193e3e097-kube-api-access-9dzxc\") pod \"481e2d44-71e6-4242-ac12-5d1193e3e097\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " Oct 31 16:14:01.614854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.614685 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-openshift-ca\") pod \"481e2d44-71e6-4242-ac12-5d1193e3e097\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " Oct 31 16:14:01.614854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.614831 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-xks-ca\") pod \"481e2d44-71e6-4242-ac12-5d1193e3e097\" (UID: \"481e2d44-71e6-4242-ac12-5d1193e3e097\") " Oct 31 16:14:01.615075 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.615051 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-openshift-ca" pod "481e2d44-71e6-4242-ac12-5d1193e3e097" (UID: "481e2d44-71e6-4242-ac12-5d1193e3e097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:14:01.616989 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.616962 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481e2d44-71e6-4242-ac12-5d1193e3e097-kube-api-access-9dzxc" pod "481e2d44-71e6-4242-ac12-5d1193e3e097" (UID: "481e2d44-71e6-4242-ac12-5d1193e3e097"). InnerVolumeSpecName "kube-api-access-9dzxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:14:01.617102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.616991 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-xks-ca" pod "481e2d44-71e6-4242-ac12-5d1193e3e097" (UID: "481e2d44-71e6-4242-ac12-5d1193e3e097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:14:01.716349 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.716312 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzxc\" (UniqueName: \"kubernetes.io/projected/481e2d44-71e6-4242-ac12-5d1193e3e097-kube-api-access-9dzxc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:14:01.716551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.716355 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:14:01.716551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:01.716373 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481e2d44-71e6-4242-ac12-5d1193e3e097-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:14:02.381553 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:02.381522 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" Oct 31 16:14:02.381553 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:02.381537 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7" event={"ID":"481e2d44-71e6-4242-ac12-5d1193e3e097","Type":"ContainerDied","Data":"c7db366a356938e41dda2703b14edb7f393faf3e11ee361659b3ffa1b1cc7113"} Oct 31 16:14:02.381782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:02.381622 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7db366a356938e41dda2703b14edb7f393faf3e11ee361659b3ffa1b1cc7113" Oct 31 16:14:09.898691 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:09.898654 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv"] Oct 31 16:14:09.899225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:09.899073 4865 cpu_manager.go:414] "containerMap: removing container" podUID="481e2d44-71e6-4242-ac12-5d1193e3e097" containerName="revoke-keys" Oct 31 16:14:09.899225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:09.899085 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481e2d44-71e6-4242-ac12-5d1193e3e097" containerName="revoke-keys" Oct 31 16:14:09.899225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:09.899148 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="481e2d44-71e6-4242-ac12-5d1193e3e097" containerName="revoke-keys" Oct 31 16:14:09.902191 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:09.902171 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:09.959023 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:09.958987 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv"] Oct 31 16:14:10.002263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.002214 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.002263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.002268 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qq5s\" (UniqueName: \"kubernetes.io/projected/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-kube-api-access-2qq5s\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.002529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.002424 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.004361 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.004341 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:14:10.004361 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.004356 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:14:10.004361 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.004363 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:14:10.004591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.004430 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:14:10.103534 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.103500 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.103721 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.103562 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.103721 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.103586 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qq5s\" (UniqueName: \"kubernetes.io/projected/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-kube-api-access-2qq5s\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.104148 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.104118 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.106137 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.106117 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.160627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.160553 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qq5s\" (UniqueName: \"kubernetes.io/projected/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-kube-api-access-2qq5s\") pod \"maas-api-revoke-sub-6b19338f-lv4cv\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.215387 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.215352 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:10.586101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:10.586069 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv"] Oct 31 16:14:10.588599 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:14:10.588568 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa8de41_a9f6_4bdf_9cbc_495a13020f84.slice/crio-2ccf971d188018e56e2dbad73fba55c8a8ae8db368882369c7af4e70129d5740 WatchSource:0}: Error finding container 2ccf971d188018e56e2dbad73fba55c8a8ae8db368882369c7af4e70129d5740: Status 404 returned error can't find the container with id 2ccf971d188018e56e2dbad73fba55c8a8ae8db368882369c7af4e70129d5740 Oct 31 16:14:11.427775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:11.427735 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" event={"ID":"0aa8de41-a9f6-4bdf-9cbc-495a13020f84","Type":"ContainerStarted","Data":"c7cd5cbf5e8f9c37bfb9728a495b539131afab1412dec1eb1cf5fa658283f3d6"} Oct 31 16:14:11.428189 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:11.427791 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" event={"ID":"0aa8de41-a9f6-4bdf-9cbc-495a13020f84","Type":"ContainerStarted","Data":"2ccf971d188018e56e2dbad73fba55c8a8ae8db368882369c7af4e70129d5740"} Oct 31 16:14:11.542977 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:11.542923 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" podStartSLOduration=2.5429065360000003 podStartE2EDuration="2.542906536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:11.540067229 +0000 UTC m=+5297.159618639" watchObservedRunningTime="2026-10-31 16:14:11.542906536 +0000 UTC m=+5297.162457979" Oct 31 16:14:12.434042 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:12.434003 4865 generic.go:356] "Generic (PLEG): container finished" podID="0aa8de41-a9f6-4bdf-9cbc-495a13020f84" containerID="c7cd5cbf5e8f9c37bfb9728a495b539131afab1412dec1eb1cf5fa658283f3d6" exitCode=0 Oct 31 16:14:12.434463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:12.434093 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" event={"ID":"0aa8de41-a9f6-4bdf-9cbc-495a13020f84","Type":"ContainerDied","Data":"c7cd5cbf5e8f9c37bfb9728a495b539131afab1412dec1eb1cf5fa658283f3d6"} Oct 31 16:14:13.574460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.574433 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:13.741408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.741372 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-xks-ca\") pod \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " Oct 31 16:14:13.741647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.741501 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-openshift-ca\") pod \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " Oct 31 16:14:13.741647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.741534 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-kube-api-access-2qq5s\" (UniqueName: \"kubernetes.io/projected/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-kube-api-access-2qq5s\") pod \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\" (UID: \"0aa8de41-a9f6-4bdf-9cbc-495a13020f84\") " Oct 31 16:14:13.741878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.741849 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-openshift-ca" pod "0aa8de41-a9f6-4bdf-9cbc-495a13020f84" (UID: "0aa8de41-a9f6-4bdf-9cbc-495a13020f84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:14:13.743817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.743789 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-xks-ca" pod "0aa8de41-a9f6-4bdf-9cbc-495a13020f84" (UID: "0aa8de41-a9f6-4bdf-9cbc-495a13020f84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:14:13.743894 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.743828 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-kube-api-access-2qq5s" pod "0aa8de41-a9f6-4bdf-9cbc-495a13020f84" (UID: "0aa8de41-a9f6-4bdf-9cbc-495a13020f84"). InnerVolumeSpecName "kube-api-access-2qq5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:14:13.842929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.842893 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:14:13.842929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.842925 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:14:13.842929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:13.842936 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qq5s\" (UniqueName: \"kubernetes.io/projected/0aa8de41-a9f6-4bdf-9cbc-495a13020f84-kube-api-access-2qq5s\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:14:14.445996 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:14.445965 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" Oct 31 16:14:14.446183 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:14.446003 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv" event={"ID":"0aa8de41-a9f6-4bdf-9cbc-495a13020f84","Type":"ContainerDied","Data":"2ccf971d188018e56e2dbad73fba55c8a8ae8db368882369c7af4e70129d5740"} Oct 31 16:14:14.446183 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:14.446095 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ccf971d188018e56e2dbad73fba55c8a8ae8db368882369c7af4e70129d5740" Oct 31 16:14:15.266153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:15.266120 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:16.761040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:16.761009 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:17.489216 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:14:17.489181 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:14:18.376010 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:18.375979 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:22.834203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:22.834165 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:14:23.993509 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:23.993445 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz"] Oct 31 16:14:23.994649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:23.994618 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0aa8de41-a9f6-4bdf-9cbc-495a13020f84" containerName="revoke-keys" Oct 31 16:14:23.994832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:23.994815 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa8de41-a9f6-4bdf-9cbc-495a13020f84" containerName="revoke-keys" Oct 31 16:14:23.999491 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:23.995094 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa8de41-a9f6-4bdf-9cbc-495a13020f84" containerName="revoke-keys" Oct 31 16:14:24.008725 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.008700 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.043987 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.043954 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:24.061958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.061930 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-6e6974a3-kserve-self-signed-certs\"" Oct 31 16:14:24.116503 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.106526 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz"] Oct 31 16:14:24.181513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.179633 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-dshm\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.181513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.179755 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0399bd36-21a2-4a20-bb14-4bc1644e4920-tls-certs\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.181513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.179802 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-tmp-dir\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.181513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.179847 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-model-cache\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.181513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.179883 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgdx\" (UniqueName: \"kubernetes.io/projected/0399bd36-21a2-4a20-bb14-4bc1644e4920-kube-api-access-tmgdx\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.181513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.179935 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-home\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.281581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.281101 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0399bd36-21a2-4a20-bb14-4bc1644e4920-tls-certs\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.281581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.281172 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-tmp-dir\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.281581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.281209 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-model-cache\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.281581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.281246 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgdx\" (UniqueName: \"kubernetes.io/projected/0399bd36-21a2-4a20-bb14-4bc1644e4920-kube-api-access-tmgdx\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.281581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.281292 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-home\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.281581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.281359 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-dshm\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.283007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.282974 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-model-cache\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.283182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.283161 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-home\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.283499 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.283454 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-tmp-dir\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.290023 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.289967 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-dshm\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.290623 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.290597 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0399bd36-21a2-4a20-bb14-4bc1644e4920-tls-certs\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.359521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.352719 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgdx\" (UniqueName: \"kubernetes.io/projected/0399bd36-21a2-4a20-bb14-4bc1644e4920-kube-api-access-tmgdx\") pod \"e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.513956 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.513679 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:24.735495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.733026 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c"] Oct 31 16:14:24.755496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.750855 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:24.794857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.790368 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a03362-f19b-4de0-8c16-df1a769218d5-tls-certs\") pod \"e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:24.794857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.790420 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhxmx\" (UniqueName: \"kubernetes.io/projected/22a03362-f19b-4de0-8c16-df1a769218d5-kube-api-access-mhxmx\") pod \"e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:24.794857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.790933 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:24.832597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.827628 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-6e6974a3-epp-sa-dockercfg-6l9w9\"" Oct 31 16:14:24.891496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.891445 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhxmx\" (UniqueName: \"kubernetes.io/projected/22a03362-f19b-4de0-8c16-df1a769218d5-kube-api-access-mhxmx\") pod \"e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:24.891707 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.891645 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a03362-f19b-4de0-8c16-df1a769218d5-tls-certs\") pod \"e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:24.912077 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.895125 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c"] Oct 31 16:14:24.989717 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:24.989677 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhxmx\" (UniqueName: \"kubernetes.io/projected/22a03362-f19b-4de0-8c16-df1a769218d5-kube-api-access-mhxmx\") pod \"e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:25.067539 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:25.067452 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a03362-f19b-4de0-8c16-df1a769218d5-tls-certs\") pod \"e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:25.068357 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:25.068334 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:25.349494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:25.337252 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz"] Oct 31 16:14:25.621549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:25.621048 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" event={"ID":"0399bd36-21a2-4a20-bb14-4bc1644e4920","Type":"ContainerStarted","Data":"f6d6df2e27a57cf7a3213e4f904f60bb9ea81adcdf81dbaf27d94528cdc1f7b3"} Oct 31 16:14:26.284849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:26.284817 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c"] Oct 31 16:14:26.285622 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:14:26.285598 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a03362_f19b_4de0_8c16_df1a769218d5.slice/crio-78fafe46ea82f946850e25933f8c597ba6ce016c36afa93207bd4be0556ac30a WatchSource:0}: Error finding container 78fafe46ea82f946850e25933f8c597ba6ce016c36afa93207bd4be0556ac30a: Status 404 returned error can't find the container with id 78fafe46ea82f946850e25933f8c597ba6ce016c36afa93207bd4be0556ac30a Oct 31 16:14:26.631152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:26.631044 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" event={"ID":"0399bd36-21a2-4a20-bb14-4bc1644e4920","Type":"ContainerStarted","Data":"340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129"} Oct 31 16:14:26.632535 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:26.632503 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" event={"ID":"22a03362-f19b-4de0-8c16-df1a769218d5","Type":"ContainerStarted","Data":"78fafe46ea82f946850e25933f8c597ba6ce016c36afa93207bd4be0556ac30a"} Oct 31 16:14:26.745943 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:26.745883 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" podStartSLOduration=3.745868757 podStartE2EDuration="3.745868757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:26.742747016 +0000 UTC m=+5312.362298428" watchObservedRunningTime="2026-10-31 16:14:26.745868757 +0000 UTC m=+5312.365420176" Oct 31 16:14:27.515852 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:27.515816 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:28.647121 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:28.647087 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" event={"ID":"22a03362-f19b-4de0-8c16-df1a769218d5","Type":"ContainerStarted","Data":"19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec"} Oct 31 16:14:28.647531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:28.647163 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:14:28.762390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:28.762310 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" podStartSLOduration=2.9579744569999997 podStartE2EDuration="4.76229181s" totalImagesPullingTime="1.804317353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:28.758717176 +0000 UTC m=+5314.378268587" watchObservedRunningTime="2026-10-31 16:14:28.76229181 +0000 UTC m=+5314.381843222" Oct 31 16:14:30.043991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:30.043955 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4"] Oct 31 16:14:30.094039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:30.094002 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51680b8c-lftk4"] Oct 31 16:14:30.495727 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:30.495688 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd82be7-4054-4dd7-95f1-d6d460cd359f" path="/var/lib/kubelet/pods/ccd82be7-4054-4dd7-95f1-d6d460cd359f/volumes" Oct 31 16:14:30.734092 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:30.734060 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:14:34.720739 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:34.720707 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:34.756495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:34.724601 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:34.985784 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:34.985689 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:14:35.024637 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:35.024594 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:14:35.024956 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:35.024923 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:14:35.608075 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:35.608042 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:42.051114 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:42.051079 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w"] Oct 31 16:14:42.100320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:42.100288 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0b964e-q5f2w"] Oct 31 16:14:42.494803 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:42.494771 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebd0091-6352-419d-a292-0fac0c76d495" path="/var/lib/kubelet/pods/6ebd0091-6352-419d-a292-0fac0c76d495/volumes" Oct 31 16:14:47.489560 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:14:47.489523 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:14:59.658889 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:14:59.658859 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:15:00.489157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:00.489127 4865 scope.go:122] "RemoveContainer" containerID="ec1fc573e87c296504b04a89dcef8b2589025783037bab4f7937061cbf6583e1" Oct 31 16:15:08.802862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:08.802779 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:15:11.652490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:11.651562 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:15:11.652490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:11.651640 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:15:15.604677 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:15.604636 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:16.861141 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:16.861105 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:17.100111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:17.100077 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:17.489879 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:15:17.489845 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:15:18.714678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:18.714647 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:34.106309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:34.106272 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:15:34.197278 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:34.197215 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz"] Oct 31 16:15:34.202150 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:34.202116 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c"] Oct 31 16:15:34.216647 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:34.216612 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:34.341686 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:34.341654 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:36.549057 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:36.549026 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:36.667760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:36.667731 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:39.658821 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:39.658783 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:15:46.896634 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:46.896599 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:15:47.489306 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:15:47.489270 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:15:48.200867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:48.200831 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:49.219916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.219871 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" podUID="0399bd36-21a2-4a20-bb14-4bc1644e4920" containerName="main" containerID="cri-o://340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129" gracePeriod=45 Oct 31 16:15:49.223625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.223593 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" containerID="cri-o://19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec" gracePeriod=45 Oct 31 16:15:49.485855 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.485789 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:15:49.574101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.573986 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-model-cache\") pod \"0399bd36-21a2-4a20-bb14-4bc1644e4920\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " Oct 31 16:15:49.574101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574048 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-home\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-home\") pod \"0399bd36-21a2-4a20-bb14-4bc1644e4920\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " Oct 31 16:15:49.574101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574079 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0399bd36-21a2-4a20-bb14-4bc1644e4920-tls-certs\" (UniqueName: \"kubernetes.io/secret/0399bd36-21a2-4a20-bb14-4bc1644e4920-tls-certs\") pod \"0399bd36-21a2-4a20-bb14-4bc1644e4920\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574125 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-tmp-dir\") pod \"0399bd36-21a2-4a20-bb14-4bc1644e4920\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574209 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0399bd36-21a2-4a20-bb14-4bc1644e4920-kube-api-access-tmgdx\" (UniqueName: \"kubernetes.io/projected/0399bd36-21a2-4a20-bb14-4bc1644e4920-kube-api-access-tmgdx\") pod \"0399bd36-21a2-4a20-bb14-4bc1644e4920\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574289 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-home" pod "0399bd36-21a2-4a20-bb14-4bc1644e4920" (UID: "0399bd36-21a2-4a20-bb14-4bc1644e4920"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574316 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-model-cache" pod "0399bd36-21a2-4a20-bb14-4bc1644e4920" (UID: "0399bd36-21a2-4a20-bb14-4bc1644e4920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574357 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-dshm\") pod \"0399bd36-21a2-4a20-bb14-4bc1644e4920\" (UID: \"0399bd36-21a2-4a20-bb14-4bc1644e4920\") " Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574500 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-tmp-dir" pod "0399bd36-21a2-4a20-bb14-4bc1644e4920" (UID: "0399bd36-21a2-4a20-bb14-4bc1644e4920"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574700 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574720 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.574736 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.576629 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0399bd36-21a2-4a20-bb14-4bc1644e4920-tls-certs" pod "0399bd36-21a2-4a20-bb14-4bc1644e4920" (UID: "0399bd36-21a2-4a20-bb14-4bc1644e4920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.577073 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-dshm" pod "0399bd36-21a2-4a20-bb14-4bc1644e4920" (UID: "0399bd36-21a2-4a20-bb14-4bc1644e4920"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:15:49.582981 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.577133 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0399bd36-21a2-4a20-bb14-4bc1644e4920-kube-api-access-tmgdx" pod "0399bd36-21a2-4a20-bb14-4bc1644e4920" (UID: "0399bd36-21a2-4a20-bb14-4bc1644e4920"). InnerVolumeSpecName "kube-api-access-tmgdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:15:49.659173 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.659140 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:15:49.675369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.675336 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0399bd36-21a2-4a20-bb14-4bc1644e4920-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:15:49.675369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.675368 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0399bd36-21a2-4a20-bb14-4bc1644e4920-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:15:49.675586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:49.675385 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmgdx\" (UniqueName: \"kubernetes.io/projected/0399bd36-21a2-4a20-bb14-4bc1644e4920-kube-api-access-tmgdx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:15:50.290307 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.290268 4865 generic.go:356] "Generic (PLEG): container finished" podID="0399bd36-21a2-4a20-bb14-4bc1644e4920" containerID="340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129" exitCode=0 Oct 31 16:15:50.290916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.290325 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" Oct 31 16:15:50.290916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.290360 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" event={"ID":"0399bd36-21a2-4a20-bb14-4bc1644e4920","Type":"ContainerDied","Data":"340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129"} Oct 31 16:15:50.290916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.290404 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz" event={"ID":"0399bd36-21a2-4a20-bb14-4bc1644e4920","Type":"ContainerDied","Data":"f6d6df2e27a57cf7a3213e4f904f60bb9ea81adcdf81dbaf27d94528cdc1f7b3"} Oct 31 16:15:50.290916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.290420 4865 scope.go:122] "RemoveContainer" containerID="340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129" Oct 31 16:15:50.300021 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.300004 4865 scope.go:122] "RemoveContainer" containerID="340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129" Oct 31 16:15:50.300340 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:15:50.300321 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129\": container with ID starting with 340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129 not found: ID does not exist" containerID="340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129" Oct 31 16:15:50.300395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.300349 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129"} err="failed to get container status \"340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129\": rpc error: code = NotFound desc = could not find container \"340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129\": container with ID starting with 340a79cce5c1cbc43c08e08604b40a8307a4ca64ae0c895fcb1edb6a751c5129 not found: ID does not exist" Oct 31 16:15:50.453463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.453428 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz"] Oct 31 16:15:50.502887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:50.502854 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-76b44dc6-nvfrz"] Oct 31 16:15:52.320915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:52.320878 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:52.494535 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:52.494501 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0399bd36-21a2-4a20-bb14-4bc1644e4920" path="/var/lib/kubelet/pods/0399bd36-21a2-4a20-bb14-4bc1644e4920/volumes" Oct 31 16:15:52.927675 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:52.927640 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:53.144224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:53.144176 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:56.406164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:56.406135 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:15:56.421022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:56.420910 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:15:57.989037 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:57.989002 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:58.144032 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:58.143997 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:58.273040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:58.272955 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:58.785038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:58.785011 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:15:58.793501 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:58.793464 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:15:59.658537 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:59.658502 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:15:59.659004 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:15:59.658623 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:16:00.511520 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:00.511492 4865 scope.go:122] "RemoveContainer" containerID="f83e6add16a1c651518c49925a53c585afde5ce36dd73c2856b9498cdd6a4e2c" Oct 31 16:16:00.520263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:00.520244 4865 scope.go:122] "RemoveContainer" containerID="7c13f323e2ef268fd72c9343f215b15d78465e3ea79f9702883d380e29f9ae66" Oct 31 16:16:01.143521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.143487 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:01.722827 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.722791 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4"] Oct 31 16:16:01.723197 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.723185 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0399bd36-21a2-4a20-bb14-4bc1644e4920" containerName="main" Oct 31 16:16:01.723245 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.723198 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0399bd36-21a2-4a20-bb14-4bc1644e4920" containerName="main" Oct 31 16:16:01.723288 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.723272 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0399bd36-21a2-4a20-bb14-4bc1644e4920" containerName="main" Oct 31 16:16:01.728187 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.728164 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:01.783074 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.783046 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4"] Oct 31 16:16:01.803532 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.803504 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:01.803705 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.803562 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4mp\" (UniqueName: \"kubernetes.io/projected/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-kube-api-access-dh4mp\") pod \"payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:01.828422 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.828393 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-14a7e5\"" Oct 31 16:16:01.828590 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.828545 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-91c1a072-dockercfg-m5scv\"" Oct 31 16:16:01.904804 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.904773 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4mp\" (UniqueName: \"kubernetes.io/projected/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-kube-api-access-dh4mp\") pod \"payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:01.904932 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.904860 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:01.905567 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.905547 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:01.961496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:01.961455 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4mp\" (UniqueName: \"kubernetes.io/projected/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-kube-api-access-dh4mp\") pod \"payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:02.041401 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:02.041339 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:02.247224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:02.247180 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:02.411998 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:02.411968 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4"] Oct 31 16:16:02.413005 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:16:02.412976 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587f5bd0_88e6_40a7_9f5d_0bdc20d48fd6.slice/crio-2b36927f28b220c85ab1c283163f1d19631ac99fd542ae10b9fc5429ecd4bafb WatchSource:0}: Error finding container 2b36927f28b220c85ab1c283163f1d19631ac99fd542ae10b9fc5429ecd4bafb: Status 404 returned error can't find the container with id 2b36927f28b220c85ab1c283163f1d19631ac99fd542ae10b9fc5429ecd4bafb Oct 31 16:16:03.394580 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:03.394547 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" event={"ID":"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6","Type":"ContainerStarted","Data":"a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101"} Oct 31 16:16:03.395011 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:03.394598 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" event={"ID":"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6","Type":"ContainerStarted","Data":"2b36927f28b220c85ab1c283163f1d19631ac99fd542ae10b9fc5429ecd4bafb"} Oct 31 16:16:03.395011 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:03.394692 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:03.509078 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:03.509018 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" podStartSLOduration=2.50900344 podStartE2EDuration="2.50900344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:03.505295639 +0000 UTC m=+5409.124847050" watchObservedRunningTime="2026-10-31 16:16:03.50900344 +0000 UTC m=+5409.128554851" Oct 31 16:16:04.238399 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:04.238363 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:04.397855 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:04.397823 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:04.513793 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:04.513707 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:09.196337 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:09.196284 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:09.658667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:09.658633 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:16:12.032619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:12.032584 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:16:14.016748 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:14.016713 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:14.403907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:14.403833 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:16:15.942673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:15.942642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:17.437669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:17.437642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:17.488813 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:16:17.488781 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:16:18.796217 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:18.796180 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:19.059292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.059203 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:19.475759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.475734 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:16:19.481149 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.481121 4865 generic.go:356] "Generic (PLEG): container finished" podID="22a03362-f19b-4de0-8c16-df1a769218d5" containerID="19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec" exitCode=0 Oct 31 16:16:19.481284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.481161 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" Oct 31 16:16:19.481284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.481204 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" event={"ID":"22a03362-f19b-4de0-8c16-df1a769218d5","Type":"ContainerDied","Data":"19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec"} Oct 31 16:16:19.481284 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.481257 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c" event={"ID":"22a03362-f19b-4de0-8c16-df1a769218d5","Type":"ContainerDied","Data":"78fafe46ea82f946850e25933f8c597ba6ce016c36afa93207bd4be0556ac30a"} Oct 31 16:16:19.481442 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.481286 4865 scope.go:122] "RemoveContainer" containerID="19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec" Oct 31 16:16:19.489911 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.489891 4865 scope.go:122] "RemoveContainer" containerID="19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec" Oct 31 16:16:19.490173 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:16:19.490151 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec\": container with ID starting with 19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec not found: ID does not exist" containerID="19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec" Oct 31 16:16:19.490282 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.490182 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec"} err="failed to get container status \"19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec\": rpc error: code = NotFound desc = could not find container \"19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec\": container with ID starting with 19c14b837d05dde4947ac7fe6aafa89dd75375264f4a131172722ab8ee529bec not found: ID does not exist" Oct 31 16:16:19.575550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.575516 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a03362-f19b-4de0-8c16-df1a769218d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/22a03362-f19b-4de0-8c16-df1a769218d5-tls-certs\") pod \"22a03362-f19b-4de0-8c16-df1a769218d5\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " Oct 31 16:16:19.575752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.575627 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a03362-f19b-4de0-8c16-df1a769218d5-kube-api-access-mhxmx\" (UniqueName: \"kubernetes.io/projected/22a03362-f19b-4de0-8c16-df1a769218d5-kube-api-access-mhxmx\") pod \"22a03362-f19b-4de0-8c16-df1a769218d5\" (UID: \"22a03362-f19b-4de0-8c16-df1a769218d5\") " Oct 31 16:16:19.577858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.577830 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a03362-f19b-4de0-8c16-df1a769218d5-kube-api-access-mhxmx" pod "22a03362-f19b-4de0-8c16-df1a769218d5" (UID: "22a03362-f19b-4de0-8c16-df1a769218d5"). InnerVolumeSpecName "kube-api-access-mhxmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:16:19.577985 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.577878 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a03362-f19b-4de0-8c16-df1a769218d5-tls-certs" pod "22a03362-f19b-4de0-8c16-df1a769218d5" (UID: "22a03362-f19b-4de0-8c16-df1a769218d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:16:19.677868 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.677796 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a03362-f19b-4de0-8c16-df1a769218d5-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:16:19.677868 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.677823 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhxmx\" (UniqueName: \"kubernetes.io/projected/22a03362-f19b-4de0-8c16-df1a769218d5-kube-api-access-mhxmx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:16:19.900841 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.900806 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c"] Oct 31 16:16:19.904258 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:19.904230 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-6e6974a3-kserve-router-scheduler-5548c487-6dg4c"] Oct 31 16:16:20.493953 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:20.493916 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" path="/var/lib/kubelet/pods/22a03362-f19b-4de0-8c16-df1a769218d5/volumes" Oct 31 16:16:22.732355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:22.732319 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:23.333912 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:23.333878 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:23.474152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:23.474116 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:23.608008 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:23.607922 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:23.932589 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:23.932498 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:26.018306 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.018270 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp"] Oct 31 16:16:26.018755 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.018690 4865 cpu_manager.go:414] "containerMap: removing container" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" Oct 31 16:16:26.018755 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.018702 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" Oct 31 16:16:26.018834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.018767 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a03362-f19b-4de0-8c16-df1a769218d5" containerName="main" Oct 31 16:16:26.023153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.023135 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.076907 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.076873 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp"] Oct 31 16:16:26.096603 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.096566 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:26.123079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.123048 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-14a7e5\"/\"openshift-service-ca.crt\"" Oct 31 16:16:26.123079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.123072 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-14a7e5\"/\"kube-root-ca.crt\"" Oct 31 16:16:26.124780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.124763 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-14a7e5\"/\"simulator-single-main-14a7e5-kserve-self-signed-certs\"" Oct 31 16:16:26.124863 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.124783 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-14a7e5\"/\"default-dockercfg-w8mq2\"" Oct 31 16:16:26.135328 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.135304 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-model-cache\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.135457 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.135350 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tmp-dir\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.135457 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.135430 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cx6f\" (UniqueName: \"kubernetes.io/projected/4e8145e0-ee3e-45a9-90b6-fa12d705710d-kube-api-access-7cx6f\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.135567 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.135529 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tls-certs\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.135615 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.135569 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-home\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.135615 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.135610 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-dshm\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.236456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236421 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-model-cache\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.236667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236513 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tmp-dir\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.236667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236570 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cx6f\" (UniqueName: \"kubernetes.io/projected/4e8145e0-ee3e-45a9-90b6-fa12d705710d-kube-api-access-7cx6f\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.236667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236610 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tls-certs\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.236667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236653 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-home\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.236918 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236712 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-dshm\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.236918 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236894 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-model-cache\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.237041 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.236947 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tmp-dir\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.237041 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.237036 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-home\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.239095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.239075 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-dshm\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.239436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.239415 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tls-certs\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.292346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.292290 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cx6f\" (UniqueName: \"kubernetes.io/projected/4e8145e0-ee3e-45a9-90b6-fa12d705710d-kube-api-access-7cx6f\") pod \"simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.335199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.335174 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:26.706073 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:26.706048 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp"] Oct 31 16:16:26.706935 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:16:26.706911 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e8145e0_ee3e_45a9_90b6_fa12d705710d.slice/crio-cf1420c51359f631d1cfefebe97d7708ab13f1c609ce59bb6cd9d3f18ff17d16 WatchSource:0}: Error finding container cf1420c51359f631d1cfefebe97d7708ab13f1c609ce59bb6cd9d3f18ff17d16: Status 404 returned error can't find the container with id cf1420c51359f631d1cfefebe97d7708ab13f1c609ce59bb6cd9d3f18ff17d16 Oct 31 16:16:27.522698 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:27.522658 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" event={"ID":"4e8145e0-ee3e-45a9-90b6-fa12d705710d","Type":"ContainerStarted","Data":"23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a"} Oct 31 16:16:27.523109 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:27.522706 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" event={"ID":"4e8145e0-ee3e-45a9-90b6-fa12d705710d","Type":"ContainerStarted","Data":"cf1420c51359f631d1cfefebe97d7708ab13f1c609ce59bb6cd9d3f18ff17d16"} Oct 31 16:16:27.635099 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:27.635040 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" podStartSLOduration=2.635022186 podStartE2EDuration="2.635022186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:27.63439218 +0000 UTC m=+5433.253943592" watchObservedRunningTime="2026-10-31 16:16:27.635022186 +0000 UTC m=+5433.254573599" Oct 31 16:16:28.336164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:28.336137 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:28.688693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:28.688615 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:30.529499 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:30.529454 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:16:33.500348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:33.500316 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:36.335586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:36.335507 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:36.347627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:36.347602 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:36.579879 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:36.579852 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:16:38.293849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:38.293811 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:43.153676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:43.153599 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:47.489664 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:16:47.489623 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:16:47.818045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:47.818012 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:52.626710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:52.626668 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:56.404751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:56.404717 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:16:57.397734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:16:57.397700 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:02.226661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:02.226617 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:07.099360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:07.099300 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:11.440289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.440253 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl"] Oct 31 16:17:11.443824 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.443808 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.499701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.499666 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl"] Oct 31 16:17:11.545329 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.545294 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:17:11.545537 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.545375 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:17:11.545537 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.545375 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:17:11.545537 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.545401 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:17:11.570750 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.570715 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.570930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.570870 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wkk\" (UniqueName: \"kubernetes.io/projected/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-kube-api-access-56wkk\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.570930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.570905 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.671798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.671761 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wkk\" (UniqueName: \"kubernetes.io/projected/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-kube-api-access-56wkk\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.671798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.671805 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.672094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.671843 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.672559 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.672531 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.674382 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.674358 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.728180 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.728146 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wkk\" (UniqueName: \"kubernetes.io/projected/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-kube-api-access-56wkk\") pod \"maas-api-revoke-sub-696a31df-sshdl\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.757514 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.757458 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:11.910232 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:11.910191 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:12.135858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:12.135829 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl"] Oct 31 16:17:12.136947 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:17:12.136924 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e9eb1c_0975_46a3_9a89_f2d68bfced7e.slice/crio-e6ec6d540e6f4c36a02db908b6d223933d1eba9cc76ab47488ef6477e7165b1f WatchSource:0}: Error finding container e6ec6d540e6f4c36a02db908b6d223933d1eba9cc76ab47488ef6477e7165b1f: Status 404 returned error can't find the container with id e6ec6d540e6f4c36a02db908b6d223933d1eba9cc76ab47488ef6477e7165b1f Oct 31 16:17:12.761567 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:12.761523 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" event={"ID":"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e","Type":"ContainerStarted","Data":"87f485dd137e32a8f376d2b60c13c1c755b2c2604641b1fd70f547fab5babbff"} Oct 31 16:17:12.762094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:12.761582 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" event={"ID":"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e","Type":"ContainerStarted","Data":"e6ec6d540e6f4c36a02db908b6d223933d1eba9cc76ab47488ef6477e7165b1f"} Oct 31 16:17:12.874660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:12.874607 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" podStartSLOduration=1.874591229 podStartE2EDuration="1.874591229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:12.872683798 +0000 UTC m=+5478.492235210" watchObservedRunningTime="2026-10-31 16:17:12.874591229 +0000 UTC m=+5478.494142641" Oct 31 16:17:13.768044 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:13.768007 4865 generic.go:356] "Generic (PLEG): container finished" podID="e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" containerID="87f485dd137e32a8f376d2b60c13c1c755b2c2604641b1fd70f547fab5babbff" exitCode=0 Oct 31 16:17:13.768463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:13.768093 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" event={"ID":"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e","Type":"ContainerDied","Data":"87f485dd137e32a8f376d2b60c13c1c755b2c2604641b1fd70f547fab5babbff"} Oct 31 16:17:14.909633 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:14.909609 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:15.006351 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.006318 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-kube-api-access-56wkk\" (UniqueName: \"kubernetes.io/projected/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-kube-api-access-56wkk\") pod \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " Oct 31 16:17:15.006565 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.006423 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-openshift-ca\") pod \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " Oct 31 16:17:15.006565 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.006499 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-xks-ca\") pod \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\" (UID: \"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e\") " Oct 31 16:17:15.006825 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.006798 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-openshift-ca" pod "e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" (UID: "e5e9eb1c-0975-46a3-9a89-f2d68bfced7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:17:15.008662 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.008639 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-xks-ca" pod "e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" (UID: "e5e9eb1c-0975-46a3-9a89-f2d68bfced7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:17:15.008746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.008696 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-kube-api-access-56wkk" pod "e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" (UID: "e5e9eb1c-0975-46a3-9a89-f2d68bfced7e"). InnerVolumeSpecName "kube-api-access-56wkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:17:15.108068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.107980 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:15.108068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.108013 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:15.108068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.108023 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56wkk\" (UniqueName: \"kubernetes.io/projected/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e-kube-api-access-56wkk\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:15.778763 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.778728 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" Oct 31 16:17:15.778763 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.778752 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl" event={"ID":"e5e9eb1c-0975-46a3-9a89-f2d68bfced7e","Type":"ContainerDied","Data":"e6ec6d540e6f4c36a02db908b6d223933d1eba9cc76ab47488ef6477e7165b1f"} Oct 31 16:17:15.779054 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:15.778800 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ec6d540e6f4c36a02db908b6d223933d1eba9cc76ab47488ef6477e7165b1f" Oct 31 16:17:16.321605 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:16.321569 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:16.715550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:16.715516 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:17.489704 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:17:17.489671 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:17:17.788599 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:17.788523 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:19.389519 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:19.389486 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:20.045958 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:20.045925 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9"] Oct 31 16:17:20.095315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:20.095284 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-czwn9"] Oct 31 16:17:20.495097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:20.495065 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64a13f5-2983-47e5-bd3b-8e5438885c59" path="/var/lib/kubelet/pods/d64a13f5-2983-47e5-bd3b-8e5438885c59/volumes" Oct 31 16:17:21.490987 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:21.490956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:22.557606 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:22.557574 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:23.032663 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:23.032607 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:24.430066 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:24.430031 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:17:26.294882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:26.294847 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:27.529006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:27.528971 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp"] Oct 31 16:17:27.623052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:27.623019 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:28.787110 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:28.787071 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:29.663093 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:29.663061 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4"] Oct 31 16:17:30.012115 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:30.012081 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:30.137982 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:30.137939 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:30.674779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:30.674747 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:34.235433 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:34.235393 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:17:34.617534 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:34.617428 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:34.664361 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:34.664324 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" podUID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" containerName="payload-processing" containerID="cri-o://a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101" gracePeriod=25 Oct 31 16:17:34.745123 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:34.745086 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:42.549597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.549537 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" podUID="4e8145e0-ee3e-45a9-90b6-fa12d705710d" containerName="main" containerID="cri-o://23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a" gracePeriod=45 Oct 31 16:17:42.569102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.569073 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" podUID="4e8145e0-ee3e-45a9-90b6-fa12d705710d" containerName="main" probeResult="failure" output="Get \"https://10.132.1.160:8000/ready\": dial tcp 10.132.1.160:8000: connect: connection refused" Oct 31 16:17:42.802150 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.802090 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:17:42.862144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862114 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8145e0-ee3e-45a9-90b6-fa12d705710d-kube-api-access-7cx6f\" (UniqueName: \"kubernetes.io/projected/4e8145e0-ee3e-45a9-90b6-fa12d705710d-kube-api-access-7cx6f\") pod \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " Oct 31 16:17:42.862312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862203 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tmp-dir\") pod \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " Oct 31 16:17:42.862312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862235 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-home\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-home\") pod \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " Oct 31 16:17:42.862312 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862302 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-dshm\") pod \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " Oct 31 16:17:42.862507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862344 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-model-cache\") pod \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " Oct 31 16:17:42.862507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862378 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tls-certs\") pod \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\" (UID: \"4e8145e0-ee3e-45a9-90b6-fa12d705710d\") " Oct 31 16:17:42.862641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862521 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tmp-dir" pod "4e8145e0-ee3e-45a9-90b6-fa12d705710d" (UID: "4e8145e0-ee3e-45a9-90b6-fa12d705710d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:17:42.862641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862589 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-home" pod "4e8145e0-ee3e-45a9-90b6-fa12d705710d" (UID: "4e8145e0-ee3e-45a9-90b6-fa12d705710d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:17:42.862751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862668 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-model-cache" pod "4e8145e0-ee3e-45a9-90b6-fa12d705710d" (UID: "4e8145e0-ee3e-45a9-90b6-fa12d705710d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:17:42.862751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862693 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:42.862751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.862710 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:42.864586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.864566 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-dshm" pod "4e8145e0-ee3e-45a9-90b6-fa12d705710d" (UID: "4e8145e0-ee3e-45a9-90b6-fa12d705710d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:17:42.864712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.864644 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tls-certs" pod "4e8145e0-ee3e-45a9-90b6-fa12d705710d" (UID: "4e8145e0-ee3e-45a9-90b6-fa12d705710d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:17:42.864712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.864704 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8145e0-ee3e-45a9-90b6-fa12d705710d-kube-api-access-7cx6f" pod "4e8145e0-ee3e-45a9-90b6-fa12d705710d" (UID: "4e8145e0-ee3e-45a9-90b6-fa12d705710d"). InnerVolumeSpecName "kube-api-access-7cx6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:17:42.915009 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.914975 4865 generic.go:356] "Generic (PLEG): container finished" podID="4e8145e0-ee3e-45a9-90b6-fa12d705710d" containerID="23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a" exitCode=0 Oct 31 16:17:42.915155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.915027 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" Oct 31 16:17:42.915155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.915058 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" event={"ID":"4e8145e0-ee3e-45a9-90b6-fa12d705710d","Type":"ContainerDied","Data":"23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a"} Oct 31 16:17:42.915155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.915108 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp" event={"ID":"4e8145e0-ee3e-45a9-90b6-fa12d705710d","Type":"ContainerDied","Data":"cf1420c51359f631d1cfefebe97d7708ab13f1c609ce59bb6cd9d3f18ff17d16"} Oct 31 16:17:42.915155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.915126 4865 scope.go:122] "RemoveContainer" containerID="23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a" Oct 31 16:17:42.927379 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.927358 4865 scope.go:122] "RemoveContainer" containerID="23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a" Oct 31 16:17:42.928767 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:17:42.928739 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a\": container with ID starting with 23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a not found: ID does not exist" containerID="23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a" Oct 31 16:17:42.928904 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.928778 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a"} err="failed to get container status \"23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a\": rpc error: code = NotFound desc = could not find container \"23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a\": container with ID starting with 23ca1f5e3c5a6efc4e06bb11e86d5197e1e33baeca5f841edad342a9d091653a not found: ID does not exist" Oct 31 16:17:42.963806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.963770 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:42.963806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.963796 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e8145e0-ee3e-45a9-90b6-fa12d705710d-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:42.963806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.963807 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8145e0-ee3e-45a9-90b6-fa12d705710d-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:42.964120 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:42.963817 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cx6f\" (UniqueName: \"kubernetes.io/projected/4e8145e0-ee3e-45a9-90b6-fa12d705710d-kube-api-access-7cx6f\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:43.079413 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:43.079324 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp"] Oct 31 16:17:43.082694 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:43.082671 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-14a7e5/simulator-single-main-14a7e5-kserve-75c467bb9c-8fmkp"] Oct 31 16:17:44.401637 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:44.401594 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" podUID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:17:44.495640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:44.495608 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8145e0-ee3e-45a9-90b6-fa12d705710d" path="/var/lib/kubelet/pods/4e8145e0-ee3e-45a9-90b6-fa12d705710d/volumes" Oct 31 16:17:47.493875 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:17:47.493840 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:17:54.401555 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.401517 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" podUID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:17:54.915964 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.915938 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:17:54.975970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.975943 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-kube-api-access-dh4mp\" (UniqueName: \"kubernetes.io/projected/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-kube-api-access-dh4mp\") pod \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " Oct 31 16:17:54.976100 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.976041 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-plugins-config-volume\") pod \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\" (UID: \"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6\") " Oct 31 16:17:54.976391 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.976371 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-plugins-config-volume" pod "587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" (UID: "587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:17:54.978438 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.978408 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-kube-api-access-dh4mp" pod "587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" (UID: "587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6"). InnerVolumeSpecName "kube-api-access-dh4mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:17:54.978591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.978430 4865 generic.go:356] "Generic (PLEG): container finished" podID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" containerID="a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101" exitCode=0 Oct 31 16:17:54.978591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.978457 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" event={"ID":"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6","Type":"ContainerDied","Data":"a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101"} Oct 31 16:17:54.978591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.978511 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" Oct 31 16:17:54.978591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.978540 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4" event={"ID":"587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6","Type":"ContainerDied","Data":"2b36927f28b220c85ab1c283163f1d19631ac99fd542ae10b9fc5429ecd4bafb"} Oct 31 16:17:54.978591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.978569 4865 scope.go:122] "RemoveContainer" containerID="a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101" Oct 31 16:17:54.990360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.990344 4865 scope.go:122] "RemoveContainer" containerID="a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101" Oct 31 16:17:54.990745 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:17:54.990725 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101\": container with ID starting with a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101 not found: ID does not exist" containerID="a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101" Oct 31 16:17:54.990831 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:54.990752 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101"} err="failed to get container status \"a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101\": rpc error: code = NotFound desc = could not find container \"a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101\": container with ID starting with a495251a2f8f2da462af780dfffdf1c56985a6c85420eeb60802816628016101 not found: ID does not exist" Oct 31 16:17:55.076994 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:55.076952 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4mp\" (UniqueName: \"kubernetes.io/projected/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-kube-api-access-dh4mp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:55.076994 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:55.076991 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:17:55.139611 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:55.139577 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4"] Oct 31 16:17:55.145874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:55.145847 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-14a7e5-7795d8658rxgn4"] Oct 31 16:17:56.495882 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:17:56.495853 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" path="/var/lib/kubelet/pods/587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6/volumes" Oct 31 16:18:00.559511 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:00.559464 4865 scope.go:122] "RemoveContainer" containerID="8c4ca0cb203bcf1033ec7097bd4c0622738b03140c93068a7a290aef05ff42cb" Oct 31 16:18:04.475184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:04.475138 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:04.622542 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:04.622493 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:04.899762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:04.899669 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:06.257640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:06.257548 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:06.668221 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:06.668134 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:07.046841 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:07.046806 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v"] Oct 31 16:18:07.095317 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:07.095288 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mn75v"] Oct 31 16:18:07.293750 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:07.293716 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:08.495839 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:08.495800 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e251abfb-45b8-472c-888f-a76561530b4b" path="/var/lib/kubelet/pods/e251abfb-45b8-472c-888f-a76561530b4b/volumes" Oct 31 16:18:10.644609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:10.644581 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:10.771987 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:10.771950 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:16.737671 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:16.737637 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:17.490067 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:18:17.490033 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:18:18.132522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:18.132488 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:19.736250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:19.736219 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:24.222884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.222849 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc"] Oct 31 16:18:24.223289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223262 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" containerName="revoke-keys" Oct 31 16:18:24.223289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223275 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" containerName="revoke-keys" Oct 31 16:18:24.223289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223287 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4e8145e0-ee3e-45a9-90b6-fa12d705710d" containerName="main" Oct 31 16:18:24.223408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223293 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8145e0-ee3e-45a9-90b6-fa12d705710d" containerName="main" Oct 31 16:18:24.223408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223322 4865 cpu_manager.go:414] "containerMap: removing container" podUID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" containerName="payload-processing" Oct 31 16:18:24.223408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223327 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" containerName="payload-processing" Oct 31 16:18:24.223408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223385 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" containerName="revoke-keys" Oct 31 16:18:24.223408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223392 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8145e0-ee3e-45a9-90b6-fa12d705710d" containerName="main" Oct 31 16:18:24.223408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.223408 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="587f5bd0-88e6-40a7-9f5d-0bdc20d48fd6" containerName="payload-processing" Oct 31 16:18:24.226403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.226386 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.250063 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.250037 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be50af-cdf5-4b76-b72c-043a39f47542-tls-certs\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.250199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.250074 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-model-cache\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.250199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.250154 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-dshm\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.250199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.250176 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-tmp-dir\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.250376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.250205 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbnd\" (UniqueName: \"kubernetes.io/projected/a0be50af-cdf5-4b76-b72c-043a39f47542-kube-api-access-mmbnd\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.250376 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.250305 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-home\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.310398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.310367 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:24.329340 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.329318 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-43e6be9b-kserve-self-signed-certs\"" Oct 31 16:18:24.332310 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.332286 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc"] Oct 31 16:18:24.350854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.350829 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbnd\" (UniqueName: \"kubernetes.io/projected/a0be50af-cdf5-4b76-b72c-043a39f47542-kube-api-access-mmbnd\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.350968 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.350866 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-home\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.350968 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.350909 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be50af-cdf5-4b76-b72c-043a39f47542-tls-certs\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.350968 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.350951 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-model-cache\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.351166 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.351014 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-dshm\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.351166 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.351046 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-tmp-dir\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.351343 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.351323 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-home\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.351395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.351333 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-model-cache\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.351395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.351371 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-tmp-dir\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.353551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.353450 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-dshm\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.353668 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.353651 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be50af-cdf5-4b76-b72c-043a39f47542-tls-certs\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.408137 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.408102 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbnd\" (UniqueName: \"kubernetes.io/projected/a0be50af-cdf5-4b76-b72c-043a39f47542-kube-api-access-mmbnd\") pod \"e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.538147 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.538069 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:24.918412 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:24.918382 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc"] Oct 31 16:18:24.919564 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:18:24.919533 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0be50af_cdf5_4b76_b72c_043a39f47542.slice/crio-9514c02ed6837e7d32fd06bca72e930b0f605b931ce9cf6977c8a3992d42bd78 WatchSource:0}: Error finding container 9514c02ed6837e7d32fd06bca72e930b0f605b931ce9cf6977c8a3992d42bd78: Status 404 returned error can't find the container with id 9514c02ed6837e7d32fd06bca72e930b0f605b931ce9cf6977c8a3992d42bd78 Oct 31 16:18:25.123785 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:25.123746 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" event={"ID":"a0be50af-cdf5-4b76-b72c-043a39f47542","Type":"ContainerStarted","Data":"67fb6a34b7f53c82609547f75891f01a04c00a9dae8b5dc17b7be9a75232b7f0"} Oct 31 16:18:25.123969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:25.123797 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" event={"ID":"a0be50af-cdf5-4b76-b72c-043a39f47542","Type":"ContainerStarted","Data":"9514c02ed6837e7d32fd06bca72e930b0f605b931ce9cf6977c8a3992d42bd78"} Oct 31 16:18:25.239403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:25.239351 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" podStartSLOduration=1.239336205 podStartE2EDuration="1.239336205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:25.237213792 +0000 UTC m=+5550.856765204" watchObservedRunningTime="2026-10-31 16:18:25.239336205 +0000 UTC m=+5550.858887616" Oct 31 16:18:25.539184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:25.539092 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:28.690065 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:28.690029 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:18:34.538923 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:34.538890 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:34.551000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:34.550977 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:35.216384 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:35.216346 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:18:42.045521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:42.045460 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4"] Oct 31 16:18:42.094667 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:42.094639 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v8pn4"] Oct 31 16:18:42.495095 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:42.495062 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ffc269-5dcd-41bb-8990-83e32b188b3d" path="/var/lib/kubelet/pods/49ffc269-5dcd-41bb-8990-83e32b188b3d/volumes" Oct 31 16:18:43.681254 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:43.681224 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:47.489741 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:18:47.489703 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:18:51.442456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.442418 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45"] Oct 31 16:18:51.446078 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.446060 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.532305 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.532271 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:51.549001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.548975 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-43e6be9b-kserve-self-signed-certs\"" Oct 31 16:18:51.552107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.552087 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45"] Oct 31 16:18:51.560319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.560300 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-dshm\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.560403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.560327 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-model-cache\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.560525 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.560504 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-tmp-dir\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.560577 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.560560 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ab192d-3955-46a4-be89-2b791db2232e-tls-certs\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.560649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.560632 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-home\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.560699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.560660 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrm7\" (UniqueName: \"kubernetes.io/projected/b7ab192d-3955-46a4-be89-2b791db2232e-kube-api-access-gzrm7\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662213 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662178 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-dshm\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662213 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662213 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-model-cache\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662280 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-tmp-dir\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662313 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ab192d-3955-46a4-be89-2b791db2232e-tls-certs\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662371 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-home\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662396 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrm7\" (UniqueName: \"kubernetes.io/projected/b7ab192d-3955-46a4-be89-2b791db2232e-kube-api-access-gzrm7\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662723 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-tmp-dir\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662747 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-model-cache\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.662905 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.662854 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-home\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.664742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.664725 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-dshm\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.664930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.664913 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ab192d-3955-46a4-be89-2b791db2232e-tls-certs\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.718778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.718749 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrm7\" (UniqueName: \"kubernetes.io/projected/b7ab192d-3955-46a4-be89-2b791db2232e-kube-api-access-gzrm7\") pod \"e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:51.758627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:51.758598 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:52.137528 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:52.137494 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45"] Oct 31 16:18:52.138319 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:18:52.138279 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7ab192d_3955_46a4_be89_2b791db2232e.slice/crio-5d9d3280c4de12731204e688d587665d9cd29bc3fbe77361d43af8115298dd28 WatchSource:0}: Error finding container 5d9d3280c4de12731204e688d587665d9cd29bc3fbe77361d43af8115298dd28: Status 404 returned error can't find the container with id 5d9d3280c4de12731204e688d587665d9cd29bc3fbe77361d43af8115298dd28 Oct 31 16:18:52.291257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:52.291220 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" event={"ID":"b7ab192d-3955-46a4-be89-2b791db2232e","Type":"ContainerStarted","Data":"ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20"} Oct 31 16:18:52.291429 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:52.291269 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" event={"ID":"b7ab192d-3955-46a4-be89-2b791db2232e","Type":"ContainerStarted","Data":"5d9d3280c4de12731204e688d587665d9cd29bc3fbe77361d43af8115298dd28"} Oct 31 16:18:52.404857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:52.404761 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" podStartSLOduration=1.404746378 podStartE2EDuration="1.404746378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:52.402483498 +0000 UTC m=+5578.022034899" watchObservedRunningTime="2026-10-31 16:18:52.404746378 +0000 UTC m=+5578.024297788" Oct 31 16:18:52.759416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:52.759385 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:18:56.213099 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:18:56.213068 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:19:00.590857 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:00.590818 4865 scope.go:122] "RemoveContainer" containerID="2093ef5733ca5f699989a7d0276f383ce1312af6dfc433b4497cf966133541c5" Oct 31 16:19:01.759250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:01.759215 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:19:01.771746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:01.771720 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:19:02.048848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:02.048744 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7"] Oct 31 16:19:02.053430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:02.053403 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-m4lx7"] Oct 31 16:19:02.388768 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:02.388692 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:19:02.494823 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:02.494792 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481e2d44-71e6-4242-ac12-5d1193e3e097" path="/var/lib/kubelet/pods/481e2d44-71e6-4242-ac12-5d1193e3e097/volumes" Oct 31 16:19:14.044256 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:14.044224 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv"] Oct 31 16:19:14.093405 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:14.093379 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lv4cv"] Oct 31 16:19:14.495184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:14.495152 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa8de41-a9f6-4bdf-9cbc-495a13020f84" path="/var/lib/kubelet/pods/0aa8de41-a9f6-4bdf-9cbc-495a13020f84/volumes" Oct 31 16:19:16.978741 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:16.978703 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:17.488891 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:19:17.488856 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:19:18.477788 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:18.477748 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:20.083420 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.083384 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:20.731026 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.730989 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4"] Oct 31 16:19:20.734706 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.734689 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.789743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.789713 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4"] Oct 31 16:19:20.835372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.835323 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:19:20.836416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.836375 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:19:20.836510 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.836396 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:19:20.836510 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.836398 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:19:20.883731 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.883703 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwfg\" (UniqueName: \"kubernetes.io/projected/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-kube-api-access-bgwfg\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.883848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.883764 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.883900 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.883850 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.984416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.984346 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.984584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.984431 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwfg\" (UniqueName: \"kubernetes.io/projected/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-kube-api-access-bgwfg\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.984584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.984493 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.985251 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.985212 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:20.986970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:20.986950 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:21.040734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:21.040701 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwfg\" (UniqueName: \"kubernetes.io/projected/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-kube-api-access-bgwfg\") pod \"maas-api-revoke-sub-452b6d7b-5pzp4\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:21.047391 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:21.047366 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:21.420196 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:21.420167 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4"] Oct 31 16:19:21.421412 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:19:21.421367 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea0922d_bf6a_4205_ab37_3e1433a7dc4a.slice/crio-49a19162e823f5f0bdd249a39398cc6ddbf2a20a33aa13ecd5627a9134733757 WatchSource:0}: Error finding container 49a19162e823f5f0bdd249a39398cc6ddbf2a20a33aa13ecd5627a9134733757: Status 404 returned error can't find the container with id 49a19162e823f5f0bdd249a39398cc6ddbf2a20a33aa13ecd5627a9134733757 Oct 31 16:19:21.473985 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:21.473952 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" event={"ID":"eea0922d-bf6a-4205-ab37-3e1433a7dc4a","Type":"ContainerStarted","Data":"49a19162e823f5f0bdd249a39398cc6ddbf2a20a33aa13ecd5627a9134733757"} Oct 31 16:19:21.935134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:21.935096 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:19:22.483100 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:22.483063 4865 generic.go:356] "Generic (PLEG): container finished" podID="eea0922d-bf6a-4205-ab37-3e1433a7dc4a" containerID="de3da22f915180c063ce3f22d8dcde11c953a1c878a5ee7c2d849d767e847b59" exitCode=0 Oct 31 16:19:22.483620 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:22.483130 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" event={"ID":"eea0922d-bf6a-4205-ab37-3e1433a7dc4a","Type":"ContainerDied","Data":"de3da22f915180c063ce3f22d8dcde11c953a1c878a5ee7c2d849d767e847b59"} Oct 31 16:19:23.692365 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.692330 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:23.819242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.818263 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-openshift-ca\") pod \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " Oct 31 16:19:23.819242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.818355 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-xks-ca\") pod \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " Oct 31 16:19:23.819242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.818411 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-kube-api-access-bgwfg\" (UniqueName: \"kubernetes.io/projected/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-kube-api-access-bgwfg\") pod \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\" (UID: \"eea0922d-bf6a-4205-ab37-3e1433a7dc4a\") " Oct 31 16:19:23.820272 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.819967 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-openshift-ca" pod "eea0922d-bf6a-4205-ab37-3e1433a7dc4a" (UID: "eea0922d-bf6a-4205-ab37-3e1433a7dc4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:19:23.836153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.836110 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-xks-ca" pod "eea0922d-bf6a-4205-ab37-3e1433a7dc4a" (UID: "eea0922d-bf6a-4205-ab37-3e1433a7dc4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:19:23.837955 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.836459 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-kube-api-access-bgwfg" pod "eea0922d-bf6a-4205-ab37-3e1433a7dc4a" (UID: "eea0922d-bf6a-4205-ab37-3e1433a7dc4a"). InnerVolumeSpecName "kube-api-access-bgwfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:19:23.935255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.935215 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:23.935568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.935546 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:23.935733 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:23.935716 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgwfg\" (UniqueName: \"kubernetes.io/projected/eea0922d-bf6a-4205-ab37-3e1433a7dc4a-kube-api-access-bgwfg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:24.535495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:24.535224 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" event={"ID":"eea0922d-bf6a-4205-ab37-3e1433a7dc4a","Type":"ContainerDied","Data":"49a19162e823f5f0bdd249a39398cc6ddbf2a20a33aa13ecd5627a9134733757"} Oct 31 16:19:24.535495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:24.535285 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a19162e823f5f0bdd249a39398cc6ddbf2a20a33aa13ecd5627a9134733757" Oct 31 16:19:24.543490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:24.539581 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4" Oct 31 16:19:26.650592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:26.650551 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:19:26.651239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:26.651204 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:19:33.536292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:33.536248 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc"] Oct 31 16:19:33.589653 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:33.589619 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:37.235496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:37.235389 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:19:38.037636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:38.037595 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45"] Oct 31 16:19:38.042389 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:38.042353 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:41.653494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:41.646602 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:19:41.653494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:41.646713 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:19:42.361818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.361780 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt"] Oct 31 16:19:42.362257 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.362231 4865 cpu_manager.go:414] "containerMap: removing container" podUID="eea0922d-bf6a-4205-ab37-3e1433a7dc4a" containerName="revoke-keys" Oct 31 16:19:42.362309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.362259 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea0922d-bf6a-4205-ab37-3e1433a7dc4a" containerName="revoke-keys" Oct 31 16:19:42.362348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.362342 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea0922d-bf6a-4205-ab37-3e1433a7dc4a" containerName="revoke-keys" Oct 31 16:19:42.365432 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.365415 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.375134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.375111 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8phs\" (UniqueName: \"kubernetes.io/projected/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-kube-api-access-j8phs\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.375238 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.375178 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.375344 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.375317 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.420545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.420516 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt"] Oct 31 16:19:42.465990 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.465949 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:19:42.465990 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.465962 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:19:42.467097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.467078 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:19:42.467159 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.467084 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:19:42.476024 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.475971 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.476144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.476095 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8phs\" (UniqueName: \"kubernetes.io/projected/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-kube-api-access-j8phs\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.476219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.476197 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.476780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.476756 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.478872 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.478854 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.532219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.532198 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8phs\" (UniqueName: \"kubernetes.io/projected/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-kube-api-access-j8phs\") pod \"maas-api-revoke-sub-3e31a506-tptnt\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:42.686869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:42.686775 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:43.092579 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:43.090867 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt"] Oct 31 16:19:43.105142 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:19:43.104573 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1fa2a6_3d9a_4a6d_b7e6_c383c2caaa1f.slice/crio-2a8a10d4a0b13f66a28b400a2c6b04838d147e99deb52a4c8b2c943d3aea6f02 WatchSource:0}: Error finding container 2a8a10d4a0b13f66a28b400a2c6b04838d147e99deb52a4c8b2c943d3aea6f02: Status 404 returned error can't find the container with id 2a8a10d4a0b13f66a28b400a2c6b04838d147e99deb52a4c8b2c943d3aea6f02 Oct 31 16:19:43.719330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:43.719087 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" event={"ID":"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f","Type":"ContainerStarted","Data":"6f5c41ccd6be6c9667847087376a438b36ad9014e15efdf2128c35f7fd8cccab"} Oct 31 16:19:43.719330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:43.719156 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" event={"ID":"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f","Type":"ContainerStarted","Data":"2a8a10d4a0b13f66a28b400a2c6b04838d147e99deb52a4c8b2c943d3aea6f02"} Oct 31 16:19:43.840055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:43.839979 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" podStartSLOduration=1.8399577150000002 podStartE2EDuration="1.839957715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:43.837968833 +0000 UTC m=+5629.457520246" watchObservedRunningTime="2026-10-31 16:19:43.839957715 +0000 UTC m=+5629.459509128" Oct 31 16:19:44.750504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:44.745508 4865 generic.go:356] "Generic (PLEG): container finished" podID="ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" containerID="6f5c41ccd6be6c9667847087376a438b36ad9014e15efdf2128c35f7fd8cccab" exitCode=0 Oct 31 16:19:44.750504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:44.745665 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" event={"ID":"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f","Type":"ContainerDied","Data":"6f5c41ccd6be6c9667847087376a438b36ad9014e15efdf2128c35f7fd8cccab"} Oct 31 16:19:46.492962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.492634 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:46.540922 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.540882 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-openshift-ca\") pod \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " Oct 31 16:19:46.541138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.540982 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-kube-api-access-j8phs\" (UniqueName: \"kubernetes.io/projected/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-kube-api-access-j8phs\") pod \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " Oct 31 16:19:46.541138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.541090 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-xks-ca\") pod \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\" (UID: \"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f\") " Oct 31 16:19:46.541318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.541287 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-openshift-ca" pod "ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" (UID: "ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:19:46.541518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.541495 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:46.543565 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.543535 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-xks-ca" pod "ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" (UID: "ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:19:46.543791 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.543768 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-kube-api-access-j8phs" pod "ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" (UID: "ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f"). InnerVolumeSpecName "kube-api-access-j8phs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:19:46.643115 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.643080 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8phs\" (UniqueName: \"kubernetes.io/projected/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-kube-api-access-j8phs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:46.643115 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.643115 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:46.788569 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.788539 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" Oct 31 16:19:46.788751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.788584 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt" event={"ID":"ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f","Type":"ContainerDied","Data":"2a8a10d4a0b13f66a28b400a2c6b04838d147e99deb52a4c8b2c943d3aea6f02"} Oct 31 16:19:46.788751 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:46.788629 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8a10d4a0b13f66a28b400a2c6b04838d147e99deb52a4c8b2c943d3aea6f02" Oct 31 16:19:47.489203 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:19:47.489169 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:19:47.600097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:47.600019 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:19:48.556507 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.556439 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" podUID="a0be50af-cdf5-4b76-b72c-043a39f47542" containerName="main" containerID="cri-o://67fb6a34b7f53c82609547f75891f01a04c00a9dae8b5dc17b7be9a75232b7f0" gracePeriod=45 Oct 31 16:19:48.799224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.799195 4865 generic.go:356] "Generic (PLEG): container finished" podID="a0be50af-cdf5-4b76-b72c-043a39f47542" containerID="67fb6a34b7f53c82609547f75891f01a04c00a9dae8b5dc17b7be9a75232b7f0" exitCode=0 Oct 31 16:19:48.799571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.799279 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" event={"ID":"a0be50af-cdf5-4b76-b72c-043a39f47542","Type":"ContainerDied","Data":"67fb6a34b7f53c82609547f75891f01a04c00a9dae8b5dc17b7be9a75232b7f0"} Oct 31 16:19:48.799571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.799331 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" event={"ID":"a0be50af-cdf5-4b76-b72c-043a39f47542","Type":"ContainerDied","Data":"9514c02ed6837e7d32fd06bca72e930b0f605b931ce9cf6977c8a3992d42bd78"} Oct 31 16:19:48.799571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.799347 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9514c02ed6837e7d32fd06bca72e930b0f605b931ce9cf6977c8a3992d42bd78" Oct 31 16:19:48.806669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.806621 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:19:48.866341 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866316 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-tmp-dir\") pod \"a0be50af-cdf5-4b76-b72c-043a39f47542\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " Oct 31 16:19:48.866462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866391 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0be50af-cdf5-4b76-b72c-043a39f47542-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be50af-cdf5-4b76-b72c-043a39f47542-tls-certs\") pod \"a0be50af-cdf5-4b76-b72c-043a39f47542\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " Oct 31 16:19:48.866554 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866458 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0be50af-cdf5-4b76-b72c-043a39f47542-kube-api-access-mmbnd\" (UniqueName: \"kubernetes.io/projected/a0be50af-cdf5-4b76-b72c-043a39f47542-kube-api-access-mmbnd\") pod \"a0be50af-cdf5-4b76-b72c-043a39f47542\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " Oct 31 16:19:48.866554 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866536 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-home\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-home\") pod \"a0be50af-cdf5-4b76-b72c-043a39f47542\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " Oct 31 16:19:48.866651 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866573 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-dshm\") pod \"a0be50af-cdf5-4b76-b72c-043a39f47542\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " Oct 31 16:19:48.866651 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866583 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-tmp-dir" pod "a0be50af-cdf5-4b76-b72c-043a39f47542" (UID: "a0be50af-cdf5-4b76-b72c-043a39f47542"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:48.866651 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866626 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-model-cache\") pod \"a0be50af-cdf5-4b76-b72c-043a39f47542\" (UID: \"a0be50af-cdf5-4b76-b72c-043a39f47542\") " Oct 31 16:19:48.866824 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866763 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-home" pod "a0be50af-cdf5-4b76-b72c-043a39f47542" (UID: "a0be50af-cdf5-4b76-b72c-043a39f47542"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:48.866972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866947 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-model-cache" pod "a0be50af-cdf5-4b76-b72c-043a39f47542" (UID: "a0be50af-cdf5-4b76-b72c-043a39f47542"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:48.866972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866965 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:48.867108 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.866979 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:48.868782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.868763 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-dshm" pod "a0be50af-cdf5-4b76-b72c-043a39f47542" (UID: "a0be50af-cdf5-4b76-b72c-043a39f47542"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:48.869207 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.869185 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0be50af-cdf5-4b76-b72c-043a39f47542-tls-certs" pod "a0be50af-cdf5-4b76-b72c-043a39f47542" (UID: "a0be50af-cdf5-4b76-b72c-043a39f47542"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:19:48.869268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.869214 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0be50af-cdf5-4b76-b72c-043a39f47542-kube-api-access-mmbnd" pod "a0be50af-cdf5-4b76-b72c-043a39f47542" (UID: "a0be50af-cdf5-4b76-b72c-043a39f47542"). InnerVolumeSpecName "kube-api-access-mmbnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:19:48.968243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.968210 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be50af-cdf5-4b76-b72c-043a39f47542-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:48.968243 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.968241 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbnd\" (UniqueName: \"kubernetes.io/projected/a0be50af-cdf5-4b76-b72c-043a39f47542-kube-api-access-mmbnd\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:48.968460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.968254 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:48.968460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:48.968267 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be50af-cdf5-4b76-b72c-043a39f47542-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:49.804301 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:49.804272 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc" Oct 31 16:19:49.967006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:49.966966 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc"] Oct 31 16:19:50.016406 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:50.016365 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-43e6be9b-kserve-55b575495c-djfhc"] Oct 31 16:19:50.496709 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:50.496666 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0be50af-cdf5-4b76-b72c-043a39f47542" path="/var/lib/kubelet/pods/a0be50af-cdf5-4b76-b72c-043a39f47542/volumes" Oct 31 16:19:53.068203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.068144 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" podUID="b7ab192d-3955-46a4-be89-2b791db2232e" containerName="main" containerID="cri-o://ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20" gracePeriod=45 Oct 31 16:19:53.324439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.324383 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:19:53.418712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.418677 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-home\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-home\") pod \"b7ab192d-3955-46a4-be89-2b791db2232e\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " Oct 31 16:19:53.418914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.418721 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-tmp-dir\") pod \"b7ab192d-3955-46a4-be89-2b791db2232e\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " Oct 31 16:19:53.418914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.418868 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-dshm\") pod \"b7ab192d-3955-46a4-be89-2b791db2232e\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " Oct 31 16:19:53.418914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.418896 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-model-cache\") pod \"b7ab192d-3955-46a4-be89-2b791db2232e\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " Oct 31 16:19:53.419119 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.418918 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ab192d-3955-46a4-be89-2b791db2232e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ab192d-3955-46a4-be89-2b791db2232e-tls-certs\") pod \"b7ab192d-3955-46a4-be89-2b791db2232e\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " Oct 31 16:19:53.419119 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.418944 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ab192d-3955-46a4-be89-2b791db2232e-kube-api-access-gzrm7\" (UniqueName: \"kubernetes.io/projected/b7ab192d-3955-46a4-be89-2b791db2232e-kube-api-access-gzrm7\") pod \"b7ab192d-3955-46a4-be89-2b791db2232e\" (UID: \"b7ab192d-3955-46a4-be89-2b791db2232e\") " Oct 31 16:19:53.419119 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.419000 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-home" pod "b7ab192d-3955-46a4-be89-2b791db2232e" (UID: "b7ab192d-3955-46a4-be89-2b791db2232e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:53.419119 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.419095 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-tmp-dir" pod "b7ab192d-3955-46a4-be89-2b791db2232e" (UID: "b7ab192d-3955-46a4-be89-2b791db2232e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:53.419289 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.419196 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-model-cache" pod "b7ab192d-3955-46a4-be89-2b791db2232e" (UID: "b7ab192d-3955-46a4-be89-2b791db2232e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:53.419358 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.419338 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:53.419421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.419358 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:53.419421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.419388 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:53.421263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.421243 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ab192d-3955-46a4-be89-2b791db2232e-kube-api-access-gzrm7" pod "b7ab192d-3955-46a4-be89-2b791db2232e" (UID: "b7ab192d-3955-46a4-be89-2b791db2232e"). InnerVolumeSpecName "kube-api-access-gzrm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:19:53.421369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.421299 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ab192d-3955-46a4-be89-2b791db2232e-tls-certs" pod "b7ab192d-3955-46a4-be89-2b791db2232e" (UID: "b7ab192d-3955-46a4-be89-2b791db2232e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:19:53.421369 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.421321 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-dshm" pod "b7ab192d-3955-46a4-be89-2b791db2232e" (UID: "b7ab192d-3955-46a4-be89-2b791db2232e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:19:53.520773 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.520740 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7ab192d-3955-46a4-be89-2b791db2232e-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:53.520773 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.520769 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7ab192d-3955-46a4-be89-2b791db2232e-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:53.520990 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.520782 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrm7\" (UniqueName: \"kubernetes.io/projected/b7ab192d-3955-46a4-be89-2b791db2232e-kube-api-access-gzrm7\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:19:53.825702 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.825666 4865 generic.go:356] "Generic (PLEG): container finished" podID="b7ab192d-3955-46a4-be89-2b791db2232e" containerID="ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20" exitCode=0 Oct 31 16:19:53.825895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.825711 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" Oct 31 16:19:53.825895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.825743 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" event={"ID":"b7ab192d-3955-46a4-be89-2b791db2232e","Type":"ContainerDied","Data":"ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20"} Oct 31 16:19:53.825895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.825779 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45" event={"ID":"b7ab192d-3955-46a4-be89-2b791db2232e","Type":"ContainerDied","Data":"5d9d3280c4de12731204e688d587665d9cd29bc3fbe77361d43af8115298dd28"} Oct 31 16:19:53.825895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.825795 4865 scope.go:122] "RemoveContainer" containerID="ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20" Oct 31 16:19:53.835329 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.835312 4865 scope.go:122] "RemoveContainer" containerID="ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20" Oct 31 16:19:53.835641 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:19:53.835622 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20\": container with ID starting with ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20 not found: ID does not exist" containerID="ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20" Oct 31 16:19:53.835708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.835654 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20"} err="failed to get container status \"ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20\": rpc error: code = NotFound desc = could not find container \"ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20\": container with ID starting with ee6fde5e8a1cdfac0a422252645b33cc7d1792baeb2764fe1b858ae399c87c20 not found: ID does not exist" Oct 31 16:19:53.988443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:53.988409 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45"] Oct 31 16:19:54.038192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:54.038163 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-43e6be9b-kserve-f74486677-b2c45"] Oct 31 16:19:54.495634 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:54.495602 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ab192d-3955-46a4-be89-2b791db2232e" path="/var/lib/kubelet/pods/b7ab192d-3955-46a4-be89-2b791db2232e/volumes" Oct 31 16:19:55.902419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:19:55.902381 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:20:00.749070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:00.749036 4865 scope.go:122] "RemoveContainer" containerID="22aec91943f8b1fe1f29191a50365bb8492a7801f146c98186cc37fcf02542b8" Oct 31 16:20:00.758227 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:00.758210 4865 scope.go:122] "RemoveContainer" containerID="c110b8356044a37c6653a3d307593c978da8f3b69e0519ecd68db795e232f649" Oct 31 16:20:00.766025 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:00.766004 4865 scope.go:122] "RemoveContainer" containerID="62c9affe2e8b3c3fe159851c0807e4e0df0ff770b5aca1c505508ebb040b0129" Oct 31 16:20:05.646274 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:05.646223 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:20:07.276083 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276048 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96"] Oct 31 16:20:07.276541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276495 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b7ab192d-3955-46a4-be89-2b791db2232e" containerName="main" Oct 31 16:20:07.276541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276508 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ab192d-3955-46a4-be89-2b791db2232e" containerName="main" Oct 31 16:20:07.276541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276524 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a0be50af-cdf5-4b76-b72c-043a39f47542" containerName="main" Oct 31 16:20:07.276541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276530 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0be50af-cdf5-4b76-b72c-043a39f47542" containerName="main" Oct 31 16:20:07.276541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276538 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" containerName="revoke-keys" Oct 31 16:20:07.276734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276548 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" containerName="revoke-keys" Oct 31 16:20:07.276734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276649 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ab192d-3955-46a4-be89-2b791db2232e" containerName="main" Oct 31 16:20:07.276734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276661 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0be50af-cdf5-4b76-b72c-043a39f47542" containerName="main" Oct 31 16:20:07.276734 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.276667 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" containerName="revoke-keys" Oct 31 16:20:07.281738 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.281707 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.335826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.335792 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96"] Oct 31 16:20:07.381037 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.381006 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:20:07.381037 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.381036 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:20:07.381037 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.381050 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:20:07.383386 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.383366 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:20:07.461171 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.461133 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.461539 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.461510 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.461791 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.461771 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6sxq\" (UniqueName: \"kubernetes.io/projected/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-kube-api-access-c6sxq\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.563356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.563272 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6sxq\" (UniqueName: \"kubernetes.io/projected/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-kube-api-access-c6sxq\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.563356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.563333 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.563672 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.563403 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.564045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.564017 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.565937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.565919 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.619807 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.619778 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6sxq\" (UniqueName: \"kubernetes.io/projected/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-kube-api-access-c6sxq\") pod \"maas-api-revoke-sub-85a9d6c4-j8k96\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:07.895910 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:07.895828 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:08.270280 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:08.270245 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96"] Oct 31 16:20:08.270974 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:20:08.270938 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ee6d8a_d017_4fca_a92e_12d5af1fe96b.slice/crio-04857c2021df2153fddd0b85583524dc9e9189fc104d8786668460ec0d7d0f9b WatchSource:0}: Error finding container 04857c2021df2153fddd0b85583524dc9e9189fc104d8786668460ec0d7d0f9b: Status 404 returned error can't find the container with id 04857c2021df2153fddd0b85583524dc9e9189fc104d8786668460ec0d7d0f9b Oct 31 16:20:08.965592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:08.965559 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" event={"ID":"46ee6d8a-d017-4fca-a92e-12d5af1fe96b","Type":"ContainerStarted","Data":"69b9e67758aec9490a9cccbf844105067ecb64a20419d040a1d36b17d2fdf430"} Oct 31 16:20:08.966011 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:08.965618 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" event={"ID":"46ee6d8a-d017-4fca-a92e-12d5af1fe96b","Type":"ContainerStarted","Data":"04857c2021df2153fddd0b85583524dc9e9189fc104d8786668460ec0d7d0f9b"} Oct 31 16:20:09.080251 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:09.080190 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" podStartSLOduration=2.08017564 podStartE2EDuration="2.08017564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:09.077149525 +0000 UTC m=+5654.696700948" watchObservedRunningTime="2026-10-31 16:20:09.08017564 +0000 UTC m=+5654.699727051" Oct 31 16:20:09.972790 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:09.972753 4865 generic.go:356] "Generic (PLEG): container finished" podID="46ee6d8a-d017-4fca-a92e-12d5af1fe96b" containerID="69b9e67758aec9490a9cccbf844105067ecb64a20419d040a1d36b17d2fdf430" exitCode=0 Oct 31 16:20:09.973193 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:09.972830 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" event={"ID":"46ee6d8a-d017-4fca-a92e-12d5af1fe96b","Type":"ContainerDied","Data":"69b9e67758aec9490a9cccbf844105067ecb64a20419d040a1d36b17d2fdf430"} Oct 31 16:20:11.114622 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.114598 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:11.195625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.195598 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-kube-api-access-c6sxq\" (UniqueName: \"kubernetes.io/projected/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-kube-api-access-c6sxq\") pod \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " Oct 31 16:20:11.195746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.195685 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-xks-ca\") pod \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " Oct 31 16:20:11.195746 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.195703 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-openshift-ca\") pod \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\" (UID: \"46ee6d8a-d017-4fca-a92e-12d5af1fe96b\") " Oct 31 16:20:11.196124 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.196100 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-openshift-ca" pod "46ee6d8a-d017-4fca-a92e-12d5af1fe96b" (UID: "46ee6d8a-d017-4fca-a92e-12d5af1fe96b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:20:11.197795 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.197771 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-kube-api-access-c6sxq" pod "46ee6d8a-d017-4fca-a92e-12d5af1fe96b" (UID: "46ee6d8a-d017-4fca-a92e-12d5af1fe96b"). InnerVolumeSpecName "kube-api-access-c6sxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:20:11.197912 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.197895 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-xks-ca" pod "46ee6d8a-d017-4fca-a92e-12d5af1fe96b" (UID: "46ee6d8a-d017-4fca-a92e-12d5af1fe96b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:20:11.296861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.296780 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6sxq\" (UniqueName: \"kubernetes.io/projected/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-kube-api-access-c6sxq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:20:11.296861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.296814 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:20:11.296861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.296829 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee6d8a-d017-4fca-a92e-12d5af1fe96b-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:20:11.984450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.984414 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" event={"ID":"46ee6d8a-d017-4fca-a92e-12d5af1fe96b","Type":"ContainerDied","Data":"04857c2021df2153fddd0b85583524dc9e9189fc104d8786668460ec0d7d0f9b"} Oct 31 16:20:11.984661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.984509 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04857c2021df2153fddd0b85583524dc9e9189fc104d8786668460ec0d7d0f9b" Oct 31 16:20:11.984661 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:11.984427 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96" Oct 31 16:20:17.319960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:17.319923 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:17.489616 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:20:17.489583 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:20:18.819130 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:18.819094 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:20.125363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:20.125325 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:20:20.422501 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:20.422407 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:23.933266 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:23.933230 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz"] Oct 31 16:20:23.933842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:23.933742 4865 cpu_manager.go:414] "containerMap: removing container" podUID="46ee6d8a-d017-4fca-a92e-12d5af1fe96b" containerName="revoke-keys" Oct 31 16:20:23.933842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:23.933757 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ee6d8a-d017-4fca-a92e-12d5af1fe96b" containerName="revoke-keys" Oct 31 16:20:23.933842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:23.933838 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ee6d8a-d017-4fca-a92e-12d5af1fe96b" containerName="revoke-keys" Oct 31 16:20:23.936718 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:23.936702 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:23.992578 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:23.992557 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 16:20:24.017994 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.017969 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65b1b83-84a3-4580-8797-de6a45a160dd-plugins-config-volume\") pod \"payload-pre-processing-6647bb8dc5-j7xxz\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:24.018168 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.018025 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwf2c\" (UniqueName: \"kubernetes.io/projected/b65b1b83-84a3-4580-8797-de6a45a160dd-kube-api-access-zwf2c\") pod \"payload-pre-processing-6647bb8dc5-j7xxz\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:24.039846 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.039825 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l4msp\"" Oct 31 16:20:24.043093 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.043069 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz"] Oct 31 16:20:24.119533 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.119505 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwf2c\" (UniqueName: \"kubernetes.io/projected/b65b1b83-84a3-4580-8797-de6a45a160dd-kube-api-access-zwf2c\") pod \"payload-pre-processing-6647bb8dc5-j7xxz\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:24.119649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.119603 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65b1b83-84a3-4580-8797-de6a45a160dd-plugins-config-volume\") pod \"payload-pre-processing-6647bb8dc5-j7xxz\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:24.120283 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.120263 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65b1b83-84a3-4580-8797-de6a45a160dd-plugins-config-volume\") pod \"payload-pre-processing-6647bb8dc5-j7xxz\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:24.178302 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.178258 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwf2c\" (UniqueName: \"kubernetes.io/projected/b65b1b83-84a3-4580-8797-de6a45a160dd-kube-api-access-zwf2c\") pod \"payload-pre-processing-6647bb8dc5-j7xxz\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:24.249370 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.249338 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:24.618626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:24.618598 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz"] Oct 31 16:20:24.619619 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:20:24.619597 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65b1b83_84a3_4580_8797_de6a45a160dd.slice/crio-cce7c1ef3122c1c9dc718a2516b5c5c5deb1640bce2f05be4d1db967ccfbbe1a WatchSource:0}: Error finding container cce7c1ef3122c1c9dc718a2516b5c5c5deb1640bce2f05be4d1db967ccfbbe1a: Status 404 returned error can't find the container with id cce7c1ef3122c1c9dc718a2516b5c5c5deb1640bce2f05be4d1db967ccfbbe1a Oct 31 16:20:25.048072 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:25.048036 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" event={"ID":"b65b1b83-84a3-4580-8797-de6a45a160dd","Type":"ContainerStarted","Data":"6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131"} Oct 31 16:20:25.048531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:25.048098 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" event={"ID":"b65b1b83-84a3-4580-8797-de6a45a160dd","Type":"ContainerStarted","Data":"cce7c1ef3122c1c9dc718a2516b5c5c5deb1640bce2f05be4d1db967ccfbbe1a"} Oct 31 16:20:25.048531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:25.048188 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:25.162395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:25.162338 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" podStartSLOduration=2.162323074 podStartE2EDuration="2.162323074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:25.160050775 +0000 UTC m=+5670.779602186" watchObservedRunningTime="2026-10-31 16:20:25.162323074 +0000 UTC m=+5670.781874485" Oct 31 16:20:36.056230 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:36.056200 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:20:38.180929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.180894 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5dd8b74fb8-dksmp"] Oct 31 16:20:38.184529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.184512 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.237649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.237616 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811f0669-afc6-4caf-8fa6-b1188fb8e23d-plugins-config-volume\") pod \"payload-processing-5dd8b74fb8-dksmp\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.237786 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.237762 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsbxw\" (UniqueName: \"kubernetes.io/projected/811f0669-afc6-4caf-8fa6-b1188fb8e23d-kube-api-access-rsbxw\") pod \"payload-processing-5dd8b74fb8-dksmp\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.239869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.239849 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5dd8b74fb8-dksmp"] Oct 31 16:20:38.338425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.338390 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsbxw\" (UniqueName: \"kubernetes.io/projected/811f0669-afc6-4caf-8fa6-b1188fb8e23d-kube-api-access-rsbxw\") pod \"payload-processing-5dd8b74fb8-dksmp\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.338622 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.338533 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811f0669-afc6-4caf-8fa6-b1188fb8e23d-plugins-config-volume\") pod \"payload-processing-5dd8b74fb8-dksmp\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.339205 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.339184 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811f0669-afc6-4caf-8fa6-b1188fb8e23d-plugins-config-volume\") pod \"payload-processing-5dd8b74fb8-dksmp\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.394439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.394411 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsbxw\" (UniqueName: \"kubernetes.io/projected/811f0669-afc6-4caf-8fa6-b1188fb8e23d-kube-api-access-rsbxw\") pod \"payload-processing-5dd8b74fb8-dksmp\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.497145 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.497122 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:38.867571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:38.867540 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5dd8b74fb8-dksmp"] Oct 31 16:20:38.869662 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:20:38.869633 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811f0669_afc6_4caf_8fa6_b1188fb8e23d.slice/crio-cf8f57f5dbb44168b723ec2969fc9a6dcf2046a0c54c46c72ca27389f56324d5 WatchSource:0}: Error finding container cf8f57f5dbb44168b723ec2969fc9a6dcf2046a0c54c46c72ca27389f56324d5: Status 404 returned error can't find the container with id cf8f57f5dbb44168b723ec2969fc9a6dcf2046a0c54c46c72ca27389f56324d5 Oct 31 16:20:39.120170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:39.120077 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" event={"ID":"811f0669-afc6-4caf-8fa6-b1188fb8e23d","Type":"ContainerStarted","Data":"127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368"} Oct 31 16:20:39.120170 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:39.120136 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" event={"ID":"811f0669-afc6-4caf-8fa6-b1188fb8e23d","Type":"ContainerStarted","Data":"cf8f57f5dbb44168b723ec2969fc9a6dcf2046a0c54c46c72ca27389f56324d5"} Oct 31 16:20:39.120421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:39.120180 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:47.489520 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:20:47.489458 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:20:50.128126 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:50.128096 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:20:50.241923 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:50.241862 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" podStartSLOduration=12.241847208 podStartE2EDuration="12.241847208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:39.281504173 +0000 UTC m=+5684.901055583" watchObservedRunningTime="2026-10-31 16:20:50.241847208 +0000 UTC m=+5695.861398618" Oct 31 16:20:52.403371 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.403335 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7744dd7796-82xn4"] Oct 31 16:20:52.406802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.406785 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:52.466724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.466694 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7744dd7796-82xn4"] Oct 31 16:20:52.555951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.555923 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn"] Oct 31 16:20:52.559534 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.559517 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:52.570277 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.570200 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa389a-4e98-487b-ab65-574e7c00176a-plugins-config-volume\") pod \"payload-processing-7744dd7796-82xn4\" (UID: \"bbfa389a-4e98-487b-ab65-574e7c00176a\") " pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:52.570415 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.570361 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pgn\" (UniqueName: \"kubernetes.io/projected/bbfa389a-4e98-487b-ab65-574e7c00176a-kube-api-access-c9pgn\") pod \"payload-processing-7744dd7796-82xn4\" (UID: \"bbfa389a-4e98-487b-ab65-574e7c00176a\") " pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:52.613944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.613918 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn"] Oct 31 16:20:52.671175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.671090 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8d6\" (UniqueName: \"kubernetes.io/projected/47f2d453-01ec-48cb-aa01-ba04ef0dbfc8-kube-api-access-5s8d6\") pod \"payload-pre-processing-7b99d75bdc-tcjgn\" (UID: \"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8\") " pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:52.671175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.671133 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa389a-4e98-487b-ab65-574e7c00176a-plugins-config-volume\") pod \"payload-processing-7744dd7796-82xn4\" (UID: \"bbfa389a-4e98-487b-ab65-574e7c00176a\") " pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:52.671423 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.671225 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f2d453-01ec-48cb-aa01-ba04ef0dbfc8-plugins-config-volume\") pod \"payload-pre-processing-7b99d75bdc-tcjgn\" (UID: \"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8\") " pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:52.671423 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.671274 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pgn\" (UniqueName: \"kubernetes.io/projected/bbfa389a-4e98-487b-ab65-574e7c00176a-kube-api-access-c9pgn\") pod \"payload-processing-7744dd7796-82xn4\" (UID: \"bbfa389a-4e98-487b-ab65-574e7c00176a\") " pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:52.671890 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.671864 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbfa389a-4e98-487b-ab65-574e7c00176a-plugins-config-volume\") pod \"payload-processing-7744dd7796-82xn4\" (UID: \"bbfa389a-4e98-487b-ab65-574e7c00176a\") " pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:52.727777 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.727751 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pgn\" (UniqueName: \"kubernetes.io/projected/bbfa389a-4e98-487b-ab65-574e7c00176a-kube-api-access-c9pgn\") pod \"payload-processing-7744dd7796-82xn4\" (UID: \"bbfa389a-4e98-487b-ab65-574e7c00176a\") " pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:52.772592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.772565 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8d6\" (UniqueName: \"kubernetes.io/projected/47f2d453-01ec-48cb-aa01-ba04ef0dbfc8-kube-api-access-5s8d6\") pod \"payload-pre-processing-7b99d75bdc-tcjgn\" (UID: \"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8\") " pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:52.772727 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.772610 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f2d453-01ec-48cb-aa01-ba04ef0dbfc8-plugins-config-volume\") pod \"payload-pre-processing-7b99d75bdc-tcjgn\" (UID: \"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8\") " pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:52.773306 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.773287 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f2d453-01ec-48cb-aa01-ba04ef0dbfc8-plugins-config-volume\") pod \"payload-pre-processing-7b99d75bdc-tcjgn\" (UID: \"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8\") " pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:52.828905 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.828882 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8d6\" (UniqueName: \"kubernetes.io/projected/47f2d453-01ec-48cb-aa01-ba04ef0dbfc8-kube-api-access-5s8d6\") pod \"payload-pre-processing-7b99d75bdc-tcjgn\" (UID: \"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8\") " pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:52.872319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:52.872297 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:53.019083 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:53.019054 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:53.246673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:53.246648 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn"] Oct 31 16:20:53.247252 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:20:53.247231 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f2d453_01ec_48cb_aa01_ba04ef0dbfc8.slice/crio-dc1da0c0897b31cf8646e0730f533b88b486362f1fcd295e51c1f59293b76761 WatchSource:0}: Error finding container dc1da0c0897b31cf8646e0730f533b88b486362f1fcd295e51c1f59293b76761: Status 404 returned error can't find the container with id dc1da0c0897b31cf8646e0730f533b88b486362f1fcd295e51c1f59293b76761 Oct 31 16:20:53.388406 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:53.388379 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7744dd7796-82xn4"] Oct 31 16:20:53.389243 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:20:53.389216 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfa389a_4e98_487b_ab65_574e7c00176a.slice/crio-8ec2c00a94729aa1eb7e71e4b71d735eb6b4dbb6c888caeb58f635adac8954bb WatchSource:0}: Error finding container 8ec2c00a94729aa1eb7e71e4b71d735eb6b4dbb6c888caeb58f635adac8954bb: Status 404 returned error can't find the container with id 8ec2c00a94729aa1eb7e71e4b71d735eb6b4dbb6c888caeb58f635adac8954bb Oct 31 16:20:54.194787 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.194751 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7744dd7796-82xn4" event={"ID":"bbfa389a-4e98-487b-ab65-574e7c00176a","Type":"ContainerStarted","Data":"485d9e376d9c135ff7ffbe56e2006b51e4070ff66f0e9503f66a239135705ebb"} Oct 31 16:20:54.195320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.194845 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7744dd7796-82xn4" event={"ID":"bbfa389a-4e98-487b-ab65-574e7c00176a","Type":"ContainerStarted","Data":"8ec2c00a94729aa1eb7e71e4b71d735eb6b4dbb6c888caeb58f635adac8954bb"} Oct 31 16:20:54.195320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.195002 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:20:54.196387 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.196358 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" event={"ID":"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8","Type":"ContainerStarted","Data":"ff3398f9cbd5dce7c69926eec2e334507480b6f411aa07fa7908cca311b1a8cc"} Oct 31 16:20:54.196522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.196408 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" event={"ID":"47f2d453-01ec-48cb-aa01-ba04ef0dbfc8","Type":"ContainerStarted","Data":"dc1da0c0897b31cf8646e0730f533b88b486362f1fcd295e51c1f59293b76761"} Oct 31 16:20:54.196522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.196431 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:20:54.308443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.308388 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7744dd7796-82xn4" podStartSLOduration=2.308375136 podStartE2EDuration="2.308375136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:54.306386926 +0000 UTC m=+5699.925938337" watchObservedRunningTime="2026-10-31 16:20:54.308375136 +0000 UTC m=+5699.927926544" Oct 31 16:20:54.421047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:54.420989 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" podStartSLOduration=2.420974457 podStartE2EDuration="2.420974457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:54.417912675 +0000 UTC m=+5700.037464084" watchObservedRunningTime="2026-10-31 16:20:54.420974457 +0000 UTC m=+5700.040525866" Oct 31 16:20:56.445796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:56.445768 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:20:56.453812 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:56.453793 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:20:58.819871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:58.819840 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:20:58.828017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:20:58.827993 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:21:00.797816 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:00.797784 4865 scope.go:122] "RemoveContainer" containerID="c7cd5cbf5e8f9c37bfb9728a495b539131afab1412dec1eb1cf5fa658283f3d6" Oct 31 16:21:05.205139 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:05.205111 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b99d75bdc-tcjgn" Oct 31 16:21:05.205573 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:05.205180 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7744dd7796-82xn4" Oct 31 16:21:05.373780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:05.373577 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz"] Oct 31 16:21:05.458051 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:05.458016 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5dd8b74fb8-dksmp"] Oct 31 16:21:08.095683 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:08.095651 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:09.121727 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:09.121691 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:10.374355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.374272 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" podUID="b65b1b83-84a3-4580-8797-de6a45a160dd" containerName="payload-pre-processing" containerID="cri-o://6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131" gracePeriod=25 Oct 31 16:21:10.459101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.459038 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" podUID="811f0669-afc6-4caf-8fa6-b1188fb8e23d" containerName="payload-processing" containerID="cri-o://127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368" gracePeriod=25 Oct 31 16:21:10.714187 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.714158 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:21:10.735377 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.735345 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65b1b83-84a3-4580-8797-de6a45a160dd-kube-api-access-zwf2c\" (UniqueName: \"kubernetes.io/projected/b65b1b83-84a3-4580-8797-de6a45a160dd-kube-api-access-zwf2c\") pod \"b65b1b83-84a3-4580-8797-de6a45a160dd\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " Oct 31 16:21:10.735613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.735427 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65b1b83-84a3-4580-8797-de6a45a160dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65b1b83-84a3-4580-8797-de6a45a160dd-plugins-config-volume\") pod \"b65b1b83-84a3-4580-8797-de6a45a160dd\" (UID: \"b65b1b83-84a3-4580-8797-de6a45a160dd\") " Oct 31 16:21:10.735822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.735799 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65b1b83-84a3-4580-8797-de6a45a160dd-plugins-config-volume" pod "b65b1b83-84a3-4580-8797-de6a45a160dd" (UID: "b65b1b83-84a3-4580-8797-de6a45a160dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:21:10.735932 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.735918 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65b1b83-84a3-4580-8797-de6a45a160dd-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:21:10.737764 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.737736 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65b1b83-84a3-4580-8797-de6a45a160dd-kube-api-access-zwf2c" pod "b65b1b83-84a3-4580-8797-de6a45a160dd" (UID: "b65b1b83-84a3-4580-8797-de6a45a160dd"). InnerVolumeSpecName "kube-api-access-zwf2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:21:10.755567 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.755542 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:21:10.837286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.837251 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811f0669-afc6-4caf-8fa6-b1188fb8e23d-kube-api-access-rsbxw\" (UniqueName: \"kubernetes.io/projected/811f0669-afc6-4caf-8fa6-b1188fb8e23d-kube-api-access-rsbxw\") pod \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " Oct 31 16:21:10.837461 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.837384 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811f0669-afc6-4caf-8fa6-b1188fb8e23d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811f0669-afc6-4caf-8fa6-b1188fb8e23d-plugins-config-volume\") pod \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\" (UID: \"811f0669-afc6-4caf-8fa6-b1188fb8e23d\") " Oct 31 16:21:10.837679 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.837665 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwf2c\" (UniqueName: \"kubernetes.io/projected/b65b1b83-84a3-4580-8797-de6a45a160dd-kube-api-access-zwf2c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:21:10.837854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.837828 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811f0669-afc6-4caf-8fa6-b1188fb8e23d-plugins-config-volume" pod "811f0669-afc6-4caf-8fa6-b1188fb8e23d" (UID: "811f0669-afc6-4caf-8fa6-b1188fb8e23d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:21:10.839701 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.839675 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811f0669-afc6-4caf-8fa6-b1188fb8e23d-kube-api-access-rsbxw" pod "811f0669-afc6-4caf-8fa6-b1188fb8e23d" (UID: "811f0669-afc6-4caf-8fa6-b1188fb8e23d"). InnerVolumeSpecName "kube-api-access-rsbxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:21:10.939172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.939074 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811f0669-afc6-4caf-8fa6-b1188fb8e23d-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:21:10.939172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:10.939114 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsbxw\" (UniqueName: \"kubernetes.io/projected/811f0669-afc6-4caf-8fa6-b1188fb8e23d-kube-api-access-rsbxw\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:21:11.175730 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.175693 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:11.279856 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.279815 4865 generic.go:356] "Generic (PLEG): container finished" podID="b65b1b83-84a3-4580-8797-de6a45a160dd" containerID="6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131" exitCode=0 Oct 31 16:21:11.280092 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.279872 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" Oct 31 16:21:11.280092 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.279906 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" event={"ID":"b65b1b83-84a3-4580-8797-de6a45a160dd","Type":"ContainerDied","Data":"6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131"} Oct 31 16:21:11.280092 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.279956 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz" event={"ID":"b65b1b83-84a3-4580-8797-de6a45a160dd","Type":"ContainerDied","Data":"cce7c1ef3122c1c9dc718a2516b5c5c5deb1640bce2f05be4d1db967ccfbbe1a"} Oct 31 16:21:11.280092 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.279979 4865 scope.go:122] "RemoveContainer" containerID="6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131" Oct 31 16:21:11.281377 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.281352 4865 generic.go:356] "Generic (PLEG): container finished" podID="811f0669-afc6-4caf-8fa6-b1188fb8e23d" containerID="127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368" exitCode=0 Oct 31 16:21:11.281705 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.281390 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" Oct 31 16:21:11.281705 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.281400 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" event={"ID":"811f0669-afc6-4caf-8fa6-b1188fb8e23d","Type":"ContainerDied","Data":"127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368"} Oct 31 16:21:11.281705 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.281434 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5dd8b74fb8-dksmp" event={"ID":"811f0669-afc6-4caf-8fa6-b1188fb8e23d","Type":"ContainerDied","Data":"cf8f57f5dbb44168b723ec2969fc9a6dcf2046a0c54c46c72ca27389f56324d5"} Oct 31 16:21:11.289801 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.289783 4865 scope.go:122] "RemoveContainer" containerID="6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131" Oct 31 16:21:11.290096 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:21:11.290072 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131\": container with ID starting with 6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131 not found: ID does not exist" containerID="6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131" Oct 31 16:21:11.290195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.290104 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131"} err="failed to get container status \"6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131\": rpc error: code = NotFound desc = could not find container \"6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131\": container with ID starting with 6954fe704451d61f412f42dc8b422bb853c5c1ee27c2916c44791ddfbcc11131 not found: ID does not exist" Oct 31 16:21:11.290195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.290122 4865 scope.go:122] "RemoveContainer" containerID="127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368" Oct 31 16:21:11.298555 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.298538 4865 scope.go:122] "RemoveContainer" containerID="127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368" Oct 31 16:21:11.298813 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:21:11.298796 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368\": container with ID starting with 127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368 not found: ID does not exist" containerID="127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368" Oct 31 16:21:11.298876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.298819 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368"} err="failed to get container status \"127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368\": rpc error: code = NotFound desc = could not find container \"127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368\": container with ID starting with 127dab33ac86bb3b258fdace6ea3b572600a2ff250b8ce01c6c449bdb30cf368 not found: ID does not exist" Oct 31 16:21:11.443393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.443359 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5dd8b74fb8-dksmp"] Oct 31 16:21:11.446070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.446047 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5dd8b74fb8-dksmp"] Oct 31 16:21:11.552862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.552779 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz"] Oct 31 16:21:11.556654 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.556629 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6647bb8dc5-j7xxz"] Oct 31 16:21:11.688625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.688591 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-brhbk"] Oct 31 16:21:11.689061 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.689042 4865 cpu_manager.go:414] "containerMap: removing container" podUID="b65b1b83-84a3-4580-8797-de6a45a160dd" containerName="payload-pre-processing" Oct 31 16:21:11.689157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.689063 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65b1b83-84a3-4580-8797-de6a45a160dd" containerName="payload-pre-processing" Oct 31 16:21:11.689157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.689105 4865 cpu_manager.go:414] "containerMap: removing container" podUID="811f0669-afc6-4caf-8fa6-b1188fb8e23d" containerName="payload-processing" Oct 31 16:21:11.689157 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.689114 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811f0669-afc6-4caf-8fa6-b1188fb8e23d" containerName="payload-processing" Oct 31 16:21:11.689335 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.689224 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65b1b83-84a3-4580-8797-de6a45a160dd" containerName="payload-pre-processing" Oct 31 16:21:11.689335 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.689245 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="811f0669-afc6-4caf-8fa6-b1188fb8e23d" containerName="payload-processing" Oct 31 16:21:11.693527 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.693505 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.746546 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.746514 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Oct 31 16:21:11.746790 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.746765 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 16:21:11.746965 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.746646 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 16:21:11.747060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.746793 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 16:21:11.747060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.746884 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-v8hfj\"" Oct 31 16:21:11.747060 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.746930 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 16:21:11.748354 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.748330 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.748451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.748380 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.748562 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.748539 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.748641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.748625 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmhx5\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-kube-api-access-kmhx5\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.748845 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.748821 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-brhbk"] Oct 31 16:21:11.849671 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.849585 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmhx5\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-kube-api-access-kmhx5\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.849671 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.849666 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.849928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.849689 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.849928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.849710 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.850403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.850378 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.850403 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.850391 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.852381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.852362 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:11.905699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:11.905673 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmhx5\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-kube-api-access-kmhx5\") pod \"payload-processing-external-model-5bcd7997-brhbk\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:12.006484 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:12.006441 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:12.383138 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:12.383109 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-brhbk"] Oct 31 16:21:12.384056 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:21:12.384026 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114170ba_1f8d_447f_8df4_e5aa46a64666.slice/crio-57656a8de83beca32de8f7ff631e8d363d49ab00fc5396b3abc06b8ee9f1f548 WatchSource:0}: Error finding container 57656a8de83beca32de8f7ff631e8d363d49ab00fc5396b3abc06b8ee9f1f548: Status 404 returned error can't find the container with id 57656a8de83beca32de8f7ff631e8d363d49ab00fc5396b3abc06b8ee9f1f548 Oct 31 16:21:12.495146 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:12.495114 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811f0669-afc6-4caf-8fa6-b1188fb8e23d" path="/var/lib/kubelet/pods/811f0669-afc6-4caf-8fa6-b1188fb8e23d/volumes" Oct 31 16:21:12.495535 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:12.495514 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65b1b83-84a3-4580-8797-de6a45a160dd" path="/var/lib/kubelet/pods/b65b1b83-84a3-4580-8797-de6a45a160dd/volumes" Oct 31 16:21:12.819860 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:12.819821 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:21:13.299085 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:13.299032 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" event={"ID":"114170ba-1f8d-447f-8df4-e5aa46a64666","Type":"ContainerStarted","Data":"cc382e5a7dd99944f08c64f4f0ffe1e70a2eb4a71a03c38cf86dec8054435b31"} Oct 31 16:21:13.299319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:13.299110 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:13.299319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:13.299137 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" event={"ID":"114170ba-1f8d-447f-8df4-e5aa46a64666","Type":"ContainerStarted","Data":"57656a8de83beca32de8f7ff631e8d363d49ab00fc5396b3abc06b8ee9f1f548"} Oct 31 16:21:13.413495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:13.413296 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" podStartSLOduration=2.413274876 podStartE2EDuration="2.413274876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:13.411188009 +0000 UTC m=+5719.030739421" watchObservedRunningTime="2026-10-31 16:21:13.413274876 +0000 UTC m=+5719.032826289" Oct 31 16:21:17.489628 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:21:17.489591 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:21:17.712278 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:17.712243 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:19.191036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:19.191005 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:20.758286 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:20.758243 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:24.324791 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:24.324754 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:21:32.518861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:32.518805 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:32.645878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:21:32.645829 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:47.492058 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:21:47.492020 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:22:15.045162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:15.045126 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl"] Oct 31 16:22:15.094389 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:15.094359 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696a31df-sshdl"] Oct 31 16:22:16.494775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:16.494733 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e9eb1c-0975-46a3-9a89-f2d68bfced7e" path="/var/lib/kubelet/pods/e5e9eb1c-0975-46a3-9a89-f2d68bfced7e/volumes" Oct 31 16:22:17.489837 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:22:17.489795 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:22:18.084724 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:18.084689 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:19.505754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:19.505715 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:21.138159 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:21.138131 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:45.377870 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:45.377829 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:47.491446 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:22:47.491406 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:22:50.947309 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:50.947274 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh"] Oct 31 16:22:50.951005 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:50.950986 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.004972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.004935 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:22:51.051594 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.051563 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.051787 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.051687 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.051787 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.051720 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbz8\" (UniqueName: \"kubernetes.io/projected/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-kube-api-access-xkbz8\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.054458 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.054421 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:22:51.054613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.054457 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:22:51.054613 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.054485 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:22:51.057640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.057618 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh"] Oct 31 16:22:51.153052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.153014 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.153052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.153054 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbz8\" (UniqueName: \"kubernetes.io/projected/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-kube-api-access-xkbz8\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.153330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.153174 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.153815 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.153787 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.155821 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.155799 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.209431 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.209404 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbz8\" (UniqueName: \"kubernetes.io/projected/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-kube-api-access-xkbz8\") pod \"maas-api-revoke-sub-e2547622-7jbjh\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.264206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.264173 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:51.634921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.634893 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh"] Oct 31 16:22:51.635963 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:22:51.635938 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aaae81_2c7f_4f6c_8f54_f9fe39f772f0.slice/crio-e9a2136d3255e41c609bf3fa7c59b46287b3ce90d5809e65d7673bb4a9503f72 WatchSource:0}: Error finding container e9a2136d3255e41c609bf3fa7c59b46287b3ce90d5809e65d7673bb4a9503f72: Status 404 returned error can't find the container with id e9a2136d3255e41c609bf3fa7c59b46287b3ce90d5809e65d7673bb4a9503f72 Oct 31 16:22:51.833916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:51.833867 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" event={"ID":"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0","Type":"ContainerStarted","Data":"e9a2136d3255e41c609bf3fa7c59b46287b3ce90d5809e65d7673bb4a9503f72"} Oct 31 16:22:52.840836 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:52.840746 4865 generic.go:356] "Generic (PLEG): container finished" podID="e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" containerID="20e416c45946eaeecf27220705153092b1f414d1d31fc1ad4853535e05cd7012" exitCode=0 Oct 31 16:22:52.841224 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:52.840833 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" event={"ID":"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0","Type":"ContainerDied","Data":"20e416c45946eaeecf27220705153092b1f414d1d31fc1ad4853535e05cd7012"} Oct 31 16:22:53.982298 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:53.982275 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:54.079128 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.079087 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-openshift-ca\") pod \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " Oct 31 16:22:54.079128 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.079131 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-kube-api-access-xkbz8\" (UniqueName: \"kubernetes.io/projected/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-kube-api-access-xkbz8\") pod \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " Oct 31 16:22:54.079318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.079169 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-xks-ca\") pod \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\" (UID: \"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0\") " Oct 31 16:22:54.079439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.079418 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-openshift-ca" pod "e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" (UID: "e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:22:54.081480 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.081438 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-xks-ca" pod "e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" (UID: "e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:22:54.081583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.081444 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-kube-api-access-xkbz8" pod "e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" (UID: "e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0"). InnerVolumeSpecName "kube-api-access-xkbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:22:54.180244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.180163 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:22:54.180244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.180199 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbz8\" (UniqueName: \"kubernetes.io/projected/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-kube-api-access-xkbz8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:22:54.180244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.180226 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:22:54.852494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.852392 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" Oct 31 16:22:54.852494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.852431 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh" event={"ID":"e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0","Type":"ContainerDied","Data":"e9a2136d3255e41c609bf3fa7c59b46287b3ce90d5809e65d7673bb4a9503f72"} Oct 31 16:22:54.852704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:22:54.852505 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9a2136d3255e41c609bf3fa7c59b46287b3ce90d5809e65d7673bb4a9503f72" Oct 31 16:23:02.448378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:02.448340 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-brhbk"] Oct 31 16:23:02.686007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:02.685966 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:02.809456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:02.809427 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:03.106514 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:03.106405 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:05.697371 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:05.697335 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:23:07.449585 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:07.449535 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" podUID="114170ba-1f8d-447f-8df4-e5aa46a64666" containerName="payload-processing" containerID="cri-o://cc382e5a7dd99944f08c64f4f0ffe1e70a2eb4a71a03c38cf86dec8054435b31" gracePeriod=25 Oct 31 16:23:07.947061 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:07.947021 4865 generic.go:356] "Generic (PLEG): container finished" podID="114170ba-1f8d-447f-8df4-e5aa46a64666" containerID="cc382e5a7dd99944f08c64f4f0ffe1e70a2eb4a71a03c38cf86dec8054435b31" exitCode=0 Oct 31 16:23:07.947378 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:07.947352 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" event={"ID":"114170ba-1f8d-447f-8df4-e5aa46a64666","Type":"ContainerDied","Data":"cc382e5a7dd99944f08c64f4f0ffe1e70a2eb4a71a03c38cf86dec8054435b31"} Oct 31 16:23:08.322762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.322707 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:23:08.323029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.322999 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:08.350492 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.324885 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:23:08.482802 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.482760 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-provider-credentials\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-provider-credentials\") pod \"114170ba-1f8d-447f-8df4-e5aa46a64666\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " Oct 31 16:23:08.532536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.501659 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-kube-api-access-kmhx5\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-kube-api-access-kmhx5\") pod \"114170ba-1f8d-447f-8df4-e5aa46a64666\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " Oct 31 16:23:08.532862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.532834 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-routing-overlay-volume\") pod \"114170ba-1f8d-447f-8df4-e5aa46a64666\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " Oct 31 16:23:08.533031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.533014 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-plugins-config-volume\") pod \"114170ba-1f8d-447f-8df4-e5aa46a64666\" (UID: \"114170ba-1f8d-447f-8df4-e5aa46a64666\") " Oct 31 16:23:08.592497 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.565725 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-plugins-config-volume" pod "114170ba-1f8d-447f-8df4-e5aa46a64666" (UID: "114170ba-1f8d-447f-8df4-e5aa46a64666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:23:08.623496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.594680 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-routing-overlay-volume" pod "114170ba-1f8d-447f-8df4-e5aa46a64666" (UID: "114170ba-1f8d-447f-8df4-e5aa46a64666"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:23:08.623496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.614609 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-kube-api-access-kmhx5" pod "114170ba-1f8d-447f-8df4-e5aa46a64666" (UID: "114170ba-1f8d-447f-8df4-e5aa46a64666"). InnerVolumeSpecName "kube-api-access-kmhx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:23:08.671550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.668050 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-provider-credentials" pod "114170ba-1f8d-447f-8df4-e5aa46a64666" (UID: "114170ba-1f8d-447f-8df4-e5aa46a64666"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:23:08.684184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.683762 4865 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-provider-credentials\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:08.684184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.683798 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmhx5\" (UniqueName: \"kubernetes.io/projected/114170ba-1f8d-447f-8df4-e5aa46a64666-kube-api-access-kmhx5\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:08.684184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.683816 4865 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-routing-overlay-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:08.684184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.683830 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114170ba-1f8d-447f-8df4-e5aa46a64666-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:08.685225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.684728 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:23:08.685225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.684807 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:08.954164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.954075 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" Oct 31 16:23:08.954164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.954104 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-brhbk" event={"ID":"114170ba-1f8d-447f-8df4-e5aa46a64666","Type":"ContainerDied","Data":"57656a8de83beca32de8f7ff631e8d363d49ab00fc5396b3abc06b8ee9f1f548"} Oct 31 16:23:08.954400 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:08.954176 4865 scope.go:122] "RemoveContainer" containerID="cc382e5a7dd99944f08c64f4f0ffe1e70a2eb4a71a03c38cf86dec8054435b31" Oct 31 16:23:09.116238 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:09.116197 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-brhbk"] Oct 31 16:23:09.170144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:09.170103 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-brhbk"] Oct 31 16:23:10.495402 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:10.495370 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114170ba-1f8d-447f-8df4-e5aa46a64666" path="/var/lib/kubelet/pods/114170ba-1f8d-447f-8df4-e5aa46a64666/volumes" Oct 31 16:23:16.034777 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:16.034747 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:23:17.504638 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:23:17.504593 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:23:18.519761 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:18.519674 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:19.938495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:19.938089 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:21.545181 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:21.545151 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:22.889221 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.889191 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr"] Oct 31 16:23:22.889677 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.889652 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" containerName="revoke-keys" Oct 31 16:23:22.889677 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.889666 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" containerName="revoke-keys" Oct 31 16:23:22.889677 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.889678 4865 cpu_manager.go:414] "containerMap: removing container" podUID="114170ba-1f8d-447f-8df4-e5aa46a64666" containerName="payload-processing" Oct 31 16:23:22.889800 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.889683 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114170ba-1f8d-447f-8df4-e5aa46a64666" containerName="payload-processing" Oct 31 16:23:22.889800 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.889769 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" containerName="revoke-keys" Oct 31 16:23:22.889800 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.889776 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="114170ba-1f8d-447f-8df4-e5aa46a64666" containerName="payload-processing" Oct 31 16:23:22.893127 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.893109 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:22.948611 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.948576 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr"] Oct 31 16:23:22.993744 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.993695 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:23:22.994271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.994221 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:23:22.994271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.994239 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:23:22.994271 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:22.994256 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:23:23.006383 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.006351 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4164b92e-d477-42df-91d6-fa530701800d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.006535 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.006518 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlqgq\" (UniqueName: \"kubernetes.io/projected/4164b92e-d477-42df-91d6-fa530701800d-kube-api-access-tlqgq\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.006607 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.006566 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4164b92e-d477-42df-91d6-fa530701800d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.107292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.107261 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlqgq\" (UniqueName: \"kubernetes.io/projected/4164b92e-d477-42df-91d6-fa530701800d-kube-api-access-tlqgq\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.107441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.107324 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4164b92e-d477-42df-91d6-fa530701800d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.107441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.107368 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4164b92e-d477-42df-91d6-fa530701800d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.108021 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.108001 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4164b92e-d477-42df-91d6-fa530701800d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.109876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.109855 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4164b92e-d477-42df-91d6-fa530701800d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.163437 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.163382 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlqgq\" (UniqueName: \"kubernetes.io/projected/4164b92e-d477-42df-91d6-fa530701800d-kube-api-access-tlqgq\") pod \"maas-api-revoke-sub-c1e271a9-7p2fr\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.205883 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.205863 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:23.587654 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:23.587629 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr"] Oct 31 16:23:24.102429 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:24.102392 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" event={"ID":"4164b92e-d477-42df-91d6-fa530701800d","Type":"ContainerStarted","Data":"bcd2fdc6798c15dbc60777079f8e023bfcca1eda208f7b0c03694601447ae3d2"} Oct 31 16:23:24.102856 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:24.102448 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" event={"ID":"4164b92e-d477-42df-91d6-fa530701800d","Type":"ContainerStarted","Data":"ce9070287f073e39200b91347f08ee581b908d55f25c284785370b69197c5c5e"} Oct 31 16:23:24.216307 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:24.216254 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" podStartSLOduration=2.216240351 podStartE2EDuration="2.216240351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:24.213772098 +0000 UTC m=+5849.833323509" watchObservedRunningTime="2026-10-31 16:23:24.216240351 +0000 UTC m=+5849.835791761" Oct 31 16:23:25.109056 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:25.109020 4865 generic.go:356] "Generic (PLEG): container finished" podID="4164b92e-d477-42df-91d6-fa530701800d" containerID="bcd2fdc6798c15dbc60777079f8e023bfcca1eda208f7b0c03694601447ae3d2" exitCode=0 Oct 31 16:23:25.109056 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:25.109058 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" event={"ID":"4164b92e-d477-42df-91d6-fa530701800d","Type":"ContainerDied","Data":"bcd2fdc6798c15dbc60777079f8e023bfcca1eda208f7b0c03694601447ae3d2"} Oct 31 16:23:26.249697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.249670 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:26.436732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.436658 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4164b92e-d477-42df-91d6-fa530701800d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4164b92e-d477-42df-91d6-fa530701800d-maas-api-openshift-ca\") pod \"4164b92e-d477-42df-91d6-fa530701800d\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " Oct 31 16:23:26.436732 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.436715 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4164b92e-d477-42df-91d6-fa530701800d-kube-api-access-tlqgq\" (UniqueName: \"kubernetes.io/projected/4164b92e-d477-42df-91d6-fa530701800d-kube-api-access-tlqgq\") pod \"4164b92e-d477-42df-91d6-fa530701800d\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " Oct 31 16:23:26.436950 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.436771 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4164b92e-d477-42df-91d6-fa530701800d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4164b92e-d477-42df-91d6-fa530701800d-maas-api-xks-ca\") pod \"4164b92e-d477-42df-91d6-fa530701800d\" (UID: \"4164b92e-d477-42df-91d6-fa530701800d\") " Oct 31 16:23:26.437023 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.437001 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4164b92e-d477-42df-91d6-fa530701800d-maas-api-openshift-ca" pod "4164b92e-d477-42df-91d6-fa530701800d" (UID: "4164b92e-d477-42df-91d6-fa530701800d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:23:26.438935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.438909 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4164b92e-d477-42df-91d6-fa530701800d-maas-api-xks-ca" pod "4164b92e-d477-42df-91d6-fa530701800d" (UID: "4164b92e-d477-42df-91d6-fa530701800d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:23:26.439055 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.438978 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4164b92e-d477-42df-91d6-fa530701800d-kube-api-access-tlqgq" pod "4164b92e-d477-42df-91d6-fa530701800d" (UID: "4164b92e-d477-42df-91d6-fa530701800d"). InnerVolumeSpecName "kube-api-access-tlqgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:23:26.538462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.538434 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4164b92e-d477-42df-91d6-fa530701800d-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:26.538462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.538461 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4164b92e-d477-42df-91d6-fa530701800d-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:26.538462 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:26.538485 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlqgq\" (UniqueName: \"kubernetes.io/projected/4164b92e-d477-42df-91d6-fa530701800d-kube-api-access-tlqgq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:27.120048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:27.120012 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" Oct 31 16:23:27.120242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:27.120046 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr" event={"ID":"4164b92e-d477-42df-91d6-fa530701800d","Type":"ContainerDied","Data":"ce9070287f073e39200b91347f08ee581b908d55f25c284785370b69197c5c5e"} Oct 31 16:23:27.120242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:27.120095 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9070287f073e39200b91347f08ee581b908d55f25c284785370b69197c5c5e" Oct 31 16:23:35.735090 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:35.735052 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:23:45.312187 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:45.312156 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:23:46.600563 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.600524 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr"] Oct 31 16:23:46.601094 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.601078 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4164b92e-d477-42df-91d6-fa530701800d" containerName="revoke-keys" Oct 31 16:23:46.601146 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.601097 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4164b92e-d477-42df-91d6-fa530701800d" containerName="revoke-keys" Oct 31 16:23:46.601233 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.601222 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4164b92e-d477-42df-91d6-fa530701800d" containerName="revoke-keys" Oct 31 16:23:46.604731 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.604710 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.658596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.658560 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr"] Oct 31 16:23:46.688390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.688361 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.688609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.688396 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.688609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.688554 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vp59\" (UniqueName: \"kubernetes.io/projected/536f940c-352e-4d05-9350-70cb16b4cf97-kube-api-access-6vp59\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.703353 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.703325 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:23:46.703490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.703369 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:23:46.703490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.703371 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:23:46.705703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.705689 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:23:46.789266 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.789237 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.789436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.789272 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.789436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.789402 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vp59\" (UniqueName: \"kubernetes.io/projected/536f940c-352e-4d05-9350-70cb16b4cf97-kube-api-access-6vp59\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.789949 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.789926 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.791872 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.791854 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.845557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.845536 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vp59\" (UniqueName: \"kubernetes.io/projected/536f940c-352e-4d05-9350-70cb16b4cf97-kube-api-access-6vp59\") pod \"maas-api-revoke-sub-2e1f5b5f-kpghr\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:46.917723 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:46.917653 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:47.291514 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:47.291463 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr"] Oct 31 16:23:47.292315 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:23:47.292287 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536f940c_352e_4d05_9350_70cb16b4cf97.slice/crio-d3dda4cad7126308c99dcf7c3b3ebb95e76f730dc06df243f7eb6c81a74cd02e WatchSource:0}: Error finding container d3dda4cad7126308c99dcf7c3b3ebb95e76f730dc06df243f7eb6c81a74cd02e: Status 404 returned error can't find the container with id d3dda4cad7126308c99dcf7c3b3ebb95e76f730dc06df243f7eb6c81a74cd02e Oct 31 16:23:47.490295 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:23:47.490263 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:23:48.282235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:48.282197 4865 generic.go:356] "Generic (PLEG): container finished" podID="536f940c-352e-4d05-9350-70cb16b4cf97" containerID="e90072ce55480b7970ae289b5b767135fb9813d5ca2bcd893a20cf0a93b0aa14" exitCode=0 Oct 31 16:23:48.282693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:48.282286 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" event={"ID":"536f940c-352e-4d05-9350-70cb16b4cf97","Type":"ContainerDied","Data":"e90072ce55480b7970ae289b5b767135fb9813d5ca2bcd893a20cf0a93b0aa14"} Oct 31 16:23:48.282693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:48.282378 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" event={"ID":"536f940c-352e-4d05-9350-70cb16b4cf97","Type":"ContainerStarted","Data":"d3dda4cad7126308c99dcf7c3b3ebb95e76f730dc06df243f7eb6c81a74cd02e"} Oct 31 16:23:49.431086 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.431063 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:49.516756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.516710 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-openshift-ca\") pod \"536f940c-352e-4d05-9350-70cb16b4cf97\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " Oct 31 16:23:49.516756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.516760 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-xks-ca\") pod \"536f940c-352e-4d05-9350-70cb16b4cf97\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " Oct 31 16:23:49.517012 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.516831 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536f940c-352e-4d05-9350-70cb16b4cf97-kube-api-access-6vp59\" (UniqueName: \"kubernetes.io/projected/536f940c-352e-4d05-9350-70cb16b4cf97-kube-api-access-6vp59\") pod \"536f940c-352e-4d05-9350-70cb16b4cf97\" (UID: \"536f940c-352e-4d05-9350-70cb16b4cf97\") " Oct 31 16:23:49.517088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.517068 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-openshift-ca" pod "536f940c-352e-4d05-9350-70cb16b4cf97" (UID: "536f940c-352e-4d05-9350-70cb16b4cf97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:23:49.519091 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.519065 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-xks-ca" pod "536f940c-352e-4d05-9350-70cb16b4cf97" (UID: "536f940c-352e-4d05-9350-70cb16b4cf97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:23:49.519211 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.519195 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536f940c-352e-4d05-9350-70cb16b4cf97-kube-api-access-6vp59" pod "536f940c-352e-4d05-9350-70cb16b4cf97" (UID: "536f940c-352e-4d05-9350-70cb16b4cf97"). InnerVolumeSpecName "kube-api-access-6vp59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:23:49.618038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.617951 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:49.618038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.617991 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536f940c-352e-4d05-9350-70cb16b4cf97-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:49.618038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:49.618005 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vp59\" (UniqueName: \"kubernetes.io/projected/536f940c-352e-4d05-9350-70cb16b4cf97-kube-api-access-6vp59\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:23:50.293699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:50.293665 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" Oct 31 16:23:50.293888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:50.293738 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr" event={"ID":"536f940c-352e-4d05-9350-70cb16b4cf97","Type":"ContainerDied","Data":"d3dda4cad7126308c99dcf7c3b3ebb95e76f730dc06df243f7eb6c81a74cd02e"} Oct 31 16:23:50.293888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:50.293784 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3dda4cad7126308c99dcf7c3b3ebb95e76f730dc06df243f7eb6c81a74cd02e" Oct 31 16:23:59.152088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.152042 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl"] Oct 31 16:23:59.152625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.152431 4865 cpu_manager.go:414] "containerMap: removing container" podUID="536f940c-352e-4d05-9350-70cb16b4cf97" containerName="revoke-keys" Oct 31 16:23:59.152625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.152442 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536f940c-352e-4d05-9350-70cb16b4cf97" containerName="revoke-keys" Oct 31 16:23:59.152625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.152557 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="536f940c-352e-4d05-9350-70cb16b4cf97" containerName="revoke-keys" Oct 31 16:23:59.155738 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.155713 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.210824 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.210783 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl"] Oct 31 16:23:59.256031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.255998 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:23:59.256031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.256000 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:23:59.256251 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.256064 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:23:59.257673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.257657 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:23:59.302729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.302698 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnt7x\" (UniqueName: \"kubernetes.io/projected/96c20188-d591-4ef0-9f6b-ce390d4b2de9-kube-api-access-vnt7x\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.302876 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.302756 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.302942 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.302908 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.404258 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.404176 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnt7x\" (UniqueName: \"kubernetes.io/projected/96c20188-d591-4ef0-9f6b-ce390d4b2de9-kube-api-access-vnt7x\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.404258 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.404220 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.404554 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.404276 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.405029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.405009 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.406968 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.406948 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.435216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.435188 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:23:59.461172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.461144 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnt7x\" (UniqueName: \"kubernetes.io/projected/96c20188-d591-4ef0-9f6b-ce390d4b2de9-kube-api-access-vnt7x\") pod \"maas-api-revoke-sub-ec843018-8r6dl\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.467878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.467849 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:23:59.853543 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:23:59.852973 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl"] Oct 31 16:24:00.346920 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:00.346887 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" event={"ID":"96c20188-d591-4ef0-9f6b-ce390d4b2de9","Type":"ContainerStarted","Data":"dab3155270a1ad1be0ea4b3fbff92b981ecda871a7ca8ee2eecbc5b16108409f"} Oct 31 16:24:00.347326 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:00.346937 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" event={"ID":"96c20188-d591-4ef0-9f6b-ce390d4b2de9","Type":"ContainerStarted","Data":"e0b1ded75ef7350cc5a5337e9fcbdb4ee1c13eeaff1ab67296c93f7e65b69d89"} Oct 31 16:24:00.461199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:00.461147 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" podStartSLOduration=1.4611322979999999 podStartE2EDuration="1.461132298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:00.45783152 +0000 UTC m=+5886.077382933" watchObservedRunningTime="2026-10-31 16:24:00.461132298 +0000 UTC m=+5886.080683709" Oct 31 16:24:00.848906 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:00.848873 4865 scope.go:122] "RemoveContainer" containerID="87f485dd137e32a8f376d2b60c13c1c755b2c2604641b1fd70f547fab5babbff" Oct 31 16:24:01.353648 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:01.353610 4865 generic.go:356] "Generic (PLEG): container finished" podID="96c20188-d591-4ef0-9f6b-ce390d4b2de9" containerID="dab3155270a1ad1be0ea4b3fbff92b981ecda871a7ca8ee2eecbc5b16108409f" exitCode=0 Oct 31 16:24:01.354049 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:01.353697 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" event={"ID":"96c20188-d591-4ef0-9f6b-ce390d4b2de9","Type":"ContainerDied","Data":"dab3155270a1ad1be0ea4b3fbff92b981ecda871a7ca8ee2eecbc5b16108409f"} Oct 31 16:24:02.499167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.499141 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:24:02.634525 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.634423 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-openshift-ca\") pod \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " Oct 31 16:24:02.634766 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.634530 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-xks-ca\") pod \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " Oct 31 16:24:02.634766 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.634675 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c20188-d591-4ef0-9f6b-ce390d4b2de9-kube-api-access-vnt7x\" (UniqueName: \"kubernetes.io/projected/96c20188-d591-4ef0-9f6b-ce390d4b2de9-kube-api-access-vnt7x\") pod \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\" (UID: \"96c20188-d591-4ef0-9f6b-ce390d4b2de9\") " Oct 31 16:24:02.634936 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.634839 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-openshift-ca" pod "96c20188-d591-4ef0-9f6b-ce390d4b2de9" (UID: "96c20188-d591-4ef0-9f6b-ce390d4b2de9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:24:02.635039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.635018 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:02.636972 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.636933 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-xks-ca" pod "96c20188-d591-4ef0-9f6b-ce390d4b2de9" (UID: "96c20188-d591-4ef0-9f6b-ce390d4b2de9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:24:02.637524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.637494 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c20188-d591-4ef0-9f6b-ce390d4b2de9-kube-api-access-vnt7x" pod "96c20188-d591-4ef0-9f6b-ce390d4b2de9" (UID: "96c20188-d591-4ef0-9f6b-ce390d4b2de9"). InnerVolumeSpecName "kube-api-access-vnt7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:24:02.735811 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.735786 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c20188-d591-4ef0-9f6b-ce390d4b2de9-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:02.735811 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:02.735811 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnt7x\" (UniqueName: \"kubernetes.io/projected/96c20188-d591-4ef0-9f6b-ce390d4b2de9-kube-api-access-vnt7x\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:03.364425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:03.364390 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" event={"ID":"96c20188-d591-4ef0-9f6b-ce390d4b2de9","Type":"ContainerDied","Data":"e0b1ded75ef7350cc5a5337e9fcbdb4ee1c13eeaff1ab67296c93f7e65b69d89"} Oct 31 16:24:03.364425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:03.364417 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl" Oct 31 16:24:03.364665 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:03.364439 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b1ded75ef7350cc5a5337e9fcbdb4ee1c13eeaff1ab67296c93f7e65b69d89" Oct 31 16:24:16.422760 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:16.422663 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:24:17.504101 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:24:17.504036 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:24:18.935045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:18.935007 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:20.029654 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:20.029615 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:24:20.030358 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:20.030313 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:20.428568 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:20.428457 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:21.792323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:21.792292 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:24.045494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:24.045434 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4"] Oct 31 16:24:24.094889 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:24.094849 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5pzp4"] Oct 31 16:24:24.495617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:24.495578 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea0922d-bf6a-4205-ab37-3e1433a7dc4a" path="/var/lib/kubelet/pods/eea0922d-bf6a-4205-ab37-3e1433a7dc4a/volumes" Oct 31 16:24:34.354808 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.354776 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv"] Oct 31 16:24:34.355504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.355396 4865 cpu_manager.go:414] "containerMap: removing container" podUID="96c20188-d591-4ef0-9f6b-ce390d4b2de9" containerName="revoke-keys" Oct 31 16:24:34.355504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.355416 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c20188-d591-4ef0-9f6b-ce390d4b2de9" containerName="revoke-keys" Oct 31 16:24:34.355670 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.355594 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c20188-d591-4ef0-9f6b-ce390d4b2de9" containerName="revoke-keys" Oct 31 16:24:34.358241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.358221 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.388650 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.388625 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.388778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.388684 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v26l\" (UniqueName: \"kubernetes.io/projected/0014dac4-f3f6-46fc-b31a-540d1c7807d7-kube-api-access-5v26l\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.388867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.388839 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.459887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.459858 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:24:34.459887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.459877 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:24:34.459887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.459896 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:24:34.460200 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.459955 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:24:34.463112 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.463089 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv"] Oct 31 16:24:34.489698 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.489673 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.489809 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.489759 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v26l\" (UniqueName: \"kubernetes.io/projected/0014dac4-f3f6-46fc-b31a-540d1c7807d7-kube-api-access-5v26l\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.489871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.489855 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.490496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.490440 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.492195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.492174 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.545692 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.545664 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v26l\" (UniqueName: \"kubernetes.io/projected/0014dac4-f3f6-46fc-b31a-540d1c7807d7-kube-api-access-5v26l\") pod \"maas-api-revoke-sub-046e1343-pntqv\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:34.669825 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:34.669759 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:35.042029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:35.041999 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv"] Oct 31 16:24:35.042828 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:24:35.042798 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0014dac4_f3f6_46fc_b31a_540d1c7807d7.slice/crio-d7b20104df1c0058df797a0ac361f0c58c7942f748f5d39c80a00a567c59f27d WatchSource:0}: Error finding container d7b20104df1c0058df797a0ac361f0c58c7942f748f5d39c80a00a567c59f27d: Status 404 returned error can't find the container with id d7b20104df1c0058df797a0ac361f0c58c7942f748f5d39c80a00a567c59f27d Oct 31 16:24:35.549332 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:35.549227 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" event={"ID":"0014dac4-f3f6-46fc-b31a-540d1c7807d7","Type":"ContainerStarted","Data":"750a56c7fcd07810af6b5866eaa4db01c4baffe5b968b09c0fd6a9e6a5c7b683"} Oct 31 16:24:35.549332 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:35.549294 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" event={"ID":"0014dac4-f3f6-46fc-b31a-540d1c7807d7","Type":"ContainerStarted","Data":"d7b20104df1c0058df797a0ac361f0c58c7942f748f5d39c80a00a567c59f27d"} Oct 31 16:24:35.665416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:35.665358 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" podStartSLOduration=1.665342684 podStartE2EDuration="1.665342684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:35.663306521 +0000 UTC m=+5921.282857937" watchObservedRunningTime="2026-10-31 16:24:35.665342684 +0000 UTC m=+5921.284894094" Oct 31 16:24:36.556276 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:36.556233 4865 generic.go:356] "Generic (PLEG): container finished" podID="0014dac4-f3f6-46fc-b31a-540d1c7807d7" containerID="750a56c7fcd07810af6b5866eaa4db01c4baffe5b968b09c0fd6a9e6a5c7b683" exitCode=0 Oct 31 16:24:36.556735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:36.556314 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" event={"ID":"0014dac4-f3f6-46fc-b31a-540d1c7807d7","Type":"ContainerDied","Data":"750a56c7fcd07810af6b5866eaa4db01c4baffe5b968b09c0fd6a9e6a5c7b683"} Oct 31 16:24:37.697918 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.697892 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:37.719413 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.719389 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0014dac4-f3f6-46fc-b31a-540d1c7807d7-kube-api-access-5v26l\" (UniqueName: \"kubernetes.io/projected/0014dac4-f3f6-46fc-b31a-540d1c7807d7-kube-api-access-5v26l\") pod \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " Oct 31 16:24:37.719610 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.719430 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-openshift-ca\") pod \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " Oct 31 16:24:37.719610 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.719490 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-xks-ca\") pod \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\" (UID: \"0014dac4-f3f6-46fc-b31a-540d1c7807d7\") " Oct 31 16:24:37.719792 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.719764 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-openshift-ca" pod "0014dac4-f3f6-46fc-b31a-540d1c7807d7" (UID: "0014dac4-f3f6-46fc-b31a-540d1c7807d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:24:37.721757 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.721735 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-xks-ca" pod "0014dac4-f3f6-46fc-b31a-540d1c7807d7" (UID: "0014dac4-f3f6-46fc-b31a-540d1c7807d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:24:37.721838 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.721765 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0014dac4-f3f6-46fc-b31a-540d1c7807d7-kube-api-access-5v26l" pod "0014dac4-f3f6-46fc-b31a-540d1c7807d7" (UID: "0014dac4-f3f6-46fc-b31a-540d1c7807d7"). InnerVolumeSpecName "kube-api-access-5v26l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:24:37.820989 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.820916 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v26l\" (UniqueName: \"kubernetes.io/projected/0014dac4-f3f6-46fc-b31a-540d1c7807d7-kube-api-access-5v26l\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:37.820989 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.820945 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:37.820989 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:37.820955 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0014dac4-f3f6-46fc-b31a-540d1c7807d7-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:38.568068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:38.568036 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" Oct 31 16:24:38.568239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:38.568080 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv" event={"ID":"0014dac4-f3f6-46fc-b31a-540d1c7807d7","Type":"ContainerDied","Data":"d7b20104df1c0058df797a0ac361f0c58c7942f748f5d39c80a00a567c59f27d"} Oct 31 16:24:38.568239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:38.568135 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7b20104df1c0058df797a0ac361f0c58c7942f748f5d39c80a00a567c59f27d" Oct 31 16:24:45.101708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.101675 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227"] Oct 31 16:24:45.102102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.102076 4865 cpu_manager.go:414] "containerMap: removing container" podUID="0014dac4-f3f6-46fc-b31a-540d1c7807d7" containerName="revoke-keys" Oct 31 16:24:45.102102 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.102087 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0014dac4-f3f6-46fc-b31a-540d1c7807d7" containerName="revoke-keys" Oct 31 16:24:45.102190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.102163 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="0014dac4-f3f6-46fc-b31a-540d1c7807d7" containerName="revoke-keys" Oct 31 16:24:45.105201 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.105184 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.161093 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.161050 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227"] Oct 31 16:24:45.186852 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.186818 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.187021 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.186953 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9lq\" (UniqueName: \"kubernetes.io/projected/1159b497-11cf-4928-8102-5b3536f9776e-kube-api-access-fk9lq\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.187083 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.187031 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.206641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.206612 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:24:45.206775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.206641 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:24:45.206775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.206641 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:24:45.206881 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.206864 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:24:45.288381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.288349 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9lq\" (UniqueName: \"kubernetes.io/projected/1159b497-11cf-4928-8102-5b3536f9776e-kube-api-access-fk9lq\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.288548 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.288445 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.288548 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.288528 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.289384 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.289356 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.290991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.290968 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.344451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.344426 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9lq\" (UniqueName: \"kubernetes.io/projected/1159b497-11cf-4928-8102-5b3536f9776e-kube-api-access-fk9lq\") pod \"maas-api-revoke-sub-d3fba056-6b227\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.417902 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.417818 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:45.788800 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:45.788775 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227"] Oct 31 16:24:45.789896 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:24:45.789865 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1159b497_11cf_4928_8102_5b3536f9776e.slice/crio-99c73ce37947c890598f4046eae157cf945ed8c749a95db2e54068003f6e8ced WatchSource:0}: Error finding container 99c73ce37947c890598f4046eae157cf945ed8c749a95db2e54068003f6e8ced: Status 404 returned error can't find the container with id 99c73ce37947c890598f4046eae157cf945ed8c749a95db2e54068003f6e8ced Oct 31 16:24:46.044649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:46.044574 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt"] Oct 31 16:24:46.047390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:46.047363 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tptnt"] Oct 31 16:24:46.494798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:46.494761 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f" path="/var/lib/kubelet/pods/ff1fa2a6-3d9a-4a6d-b7e6-c383c2caaa1f/volumes" Oct 31 16:24:46.608735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:46.608705 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" event={"ID":"1159b497-11cf-4928-8102-5b3536f9776e","Type":"ContainerStarted","Data":"0e7b5e583050305ce564d8d54604292789331a0f7689aebc3aa31a4303d31ae6"} Oct 31 16:24:46.608991 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:46.608750 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" event={"ID":"1159b497-11cf-4928-8102-5b3536f9776e","Type":"ContainerStarted","Data":"99c73ce37947c890598f4046eae157cf945ed8c749a95db2e54068003f6e8ced"} Oct 31 16:24:46.722265 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:46.722214 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" podStartSLOduration=1.722199484 podStartE2EDuration="1.722199484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:46.720696113 +0000 UTC m=+5932.340247524" watchObservedRunningTime="2026-10-31 16:24:46.722199484 +0000 UTC m=+5932.341750895" Oct 31 16:24:47.209172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:47.209142 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:24:47.489499 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:24:47.489390 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:24:47.617617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:47.617568 4865 generic.go:356] "Generic (PLEG): container finished" podID="1159b497-11cf-4928-8102-5b3536f9776e" containerID="0e7b5e583050305ce564d8d54604292789331a0f7689aebc3aa31a4303d31ae6" exitCode=0 Oct 31 16:24:47.618165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:47.617676 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" event={"ID":"1159b497-11cf-4928-8102-5b3536f9776e","Type":"ContainerDied","Data":"0e7b5e583050305ce564d8d54604292789331a0f7689aebc3aa31a4303d31ae6"} Oct 31 16:24:49.120430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.120054 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:49.202045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.201941 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-xks-ca\") pod \"1159b497-11cf-4928-8102-5b3536f9776e\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " Oct 31 16:24:49.202373 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.202327 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1159b497-11cf-4928-8102-5b3536f9776e-kube-api-access-fk9lq\" (UniqueName: \"kubernetes.io/projected/1159b497-11cf-4928-8102-5b3536f9776e-kube-api-access-fk9lq\") pod \"1159b497-11cf-4928-8102-5b3536f9776e\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " Oct 31 16:24:49.202700 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.202664 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-openshift-ca\") pod \"1159b497-11cf-4928-8102-5b3536f9776e\" (UID: \"1159b497-11cf-4928-8102-5b3536f9776e\") " Oct 31 16:24:49.203768 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.203742 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-openshift-ca" pod "1159b497-11cf-4928-8102-5b3536f9776e" (UID: "1159b497-11cf-4928-8102-5b3536f9776e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:24:49.212056 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.212023 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1159b497-11cf-4928-8102-5b3536f9776e-kube-api-access-fk9lq" pod "1159b497-11cf-4928-8102-5b3536f9776e" (UID: "1159b497-11cf-4928-8102-5b3536f9776e"). InnerVolumeSpecName "kube-api-access-fk9lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:24:49.220103 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.220048 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-xks-ca" pod "1159b497-11cf-4928-8102-5b3536f9776e" (UID: "1159b497-11cf-4928-8102-5b3536f9776e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:24:49.306261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.306152 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:49.306261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.306198 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1159b497-11cf-4928-8102-5b3536f9776e-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:49.306261 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.306216 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9lq\" (UniqueName: \"kubernetes.io/projected/1159b497-11cf-4928-8102-5b3536f9776e-kube-api-access-fk9lq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:24:49.663633 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.659463 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" Oct 31 16:24:49.663868 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.663720 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227" event={"ID":"1159b497-11cf-4928-8102-5b3536f9776e","Type":"ContainerDied","Data":"99c73ce37947c890598f4046eae157cf945ed8c749a95db2e54068003f6e8ced"} Oct 31 16:24:49.663868 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:49.663779 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99c73ce37947c890598f4046eae157cf945ed8c749a95db2e54068003f6e8ced" Oct 31 16:24:50.037696 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:50.037649 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:24:50.038062 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:50.038012 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:53.702088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:53.702050 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:53.823313 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:53.823278 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:57.827420 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:24:57.827384 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:25:00.873550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:00.873513 4865 scope.go:122] "RemoveContainer" containerID="67fb6a34b7f53c82609547f75891f01a04c00a9dae8b5dc17b7be9a75232b7f0" Oct 31 16:25:10.024948 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:10.024910 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:25:12.045951 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:12.045912 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96"] Oct 31 16:25:12.095164 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:12.095125 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9d6c4-j8k96"] Oct 31 16:25:12.495736 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:12.495694 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ee6d8a-d017-4fca-a92e-12d5af1fe96b" path="/var/lib/kubelet/pods/46ee6d8a-d017-4fca-a92e-12d5af1fe96b/volumes" Oct 31 16:25:17.488634 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:25:17.488597 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:25:19.280842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.280807 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:19.804548 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.804511 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4"] Oct 31 16:25:19.804914 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.804902 4865 cpu_manager.go:414] "containerMap: removing container" podUID="1159b497-11cf-4928-8102-5b3536f9776e" containerName="revoke-keys" Oct 31 16:25:19.804961 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.804915 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1159b497-11cf-4928-8102-5b3536f9776e" containerName="revoke-keys" Oct 31 16:25:19.804999 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.804982 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="1159b497-11cf-4928-8102-5b3536f9776e" containerName="revoke-keys" Oct 31 16:25:19.808040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.808025 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:19.863884 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.863851 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4"] Oct 31 16:25:19.909722 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.909699 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:25:19.909892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.909718 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:25:19.909892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.909721 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:25:19.909892 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.909721 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:25:19.930887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.930863 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:19.931038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.930893 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:19.931038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:19.930929 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xwq\" (UniqueName: \"kubernetes.io/projected/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-kube-api-access-d2xwq\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.032130 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.032101 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.032130 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.032132 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.032339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.032175 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xwq\" (UniqueName: \"kubernetes.io/projected/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-kube-api-access-d2xwq\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.032720 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.032702 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.034660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.034635 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.088527 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.088449 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xwq\" (UniqueName: \"kubernetes.io/projected/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-kube-api-access-d2xwq\") pod \"maas-api-revoke-sub-ae594979-wlfm4\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.120615 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.120592 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:20.492089 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:25:20.492060 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8630f5c_f75e_4a8d_b8ef_4d1edf83a8c1.slice/crio-079ad82ae146a4569908225b21d39b041d6d8c37a617eb571a376177e2348224 WatchSource:0}: Error finding container 079ad82ae146a4569908225b21d39b041d6d8c37a617eb571a376177e2348224: Status 404 returned error can't find the container with id 079ad82ae146a4569908225b21d39b041d6d8c37a617eb571a376177e2348224 Oct 31 16:25:20.495116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.495089 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4"] Oct 31 16:25:20.795360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.795291 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:20.951693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.951659 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" event={"ID":"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1","Type":"ContainerStarted","Data":"34bd4387ad5bc2bc424d85eb280625f08900bb30b7b22d2d1165101d1abf407f"} Oct 31 16:25:20.951863 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:20.951710 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" event={"ID":"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1","Type":"ContainerStarted","Data":"079ad82ae146a4569908225b21d39b041d6d8c37a617eb571a376177e2348224"} Oct 31 16:25:21.065343 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:21.065247 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" podStartSLOduration=2.065232422 podStartE2EDuration="2.065232422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:21.063912474 +0000 UTC m=+5966.683463886" watchObservedRunningTime="2026-10-31 16:25:21.065232422 +0000 UTC m=+5966.684783831" Oct 31 16:25:21.958394 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:21.958363 4865 generic.go:356] "Generic (PLEG): container finished" podID="e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" containerID="34bd4387ad5bc2bc424d85eb280625f08900bb30b7b22d2d1165101d1abf407f" exitCode=0 Oct 31 16:25:21.958826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:21.958420 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" event={"ID":"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1","Type":"ContainerDied","Data":"34bd4387ad5bc2bc424d85eb280625f08900bb30b7b22d2d1165101d1abf407f"} Oct 31 16:25:22.130591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:22.130553 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:23.101619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.101594 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:23.160849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.160822 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-openshift-ca\") pod \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " Oct 31 16:25:23.161050 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.160876 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-kube-api-access-d2xwq\" (UniqueName: \"kubernetes.io/projected/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-kube-api-access-d2xwq\") pod \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " Oct 31 16:25:23.161050 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.160925 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-xks-ca\") pod \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\" (UID: \"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1\") " Oct 31 16:25:23.161255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.161227 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-openshift-ca" pod "e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" (UID: "e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:25:23.161567 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.161344 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:25:23.164207 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.164180 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-xks-ca" pod "e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" (UID: "e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:25:23.165615 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.165584 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-kube-api-access-d2xwq" pod "e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" (UID: "e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1"). InnerVolumeSpecName "kube-api-access-d2xwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:25:23.261976 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.261897 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xwq\" (UniqueName: \"kubernetes.io/projected/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-kube-api-access-d2xwq\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:25:23.261976 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.261928 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:25:23.970700 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.970664 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" event={"ID":"e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1","Type":"ContainerDied","Data":"079ad82ae146a4569908225b21d39b041d6d8c37a617eb571a376177e2348224"} Oct 31 16:25:23.970913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.970727 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="079ad82ae146a4569908225b21d39b041d6d8c37a617eb571a376177e2348224" Oct 31 16:25:23.970913 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:23.970671 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4" Oct 31 16:25:32.708103 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:32.708068 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:25:38.554240 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.554002 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw"] Oct 31 16:25:38.555659 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.555637 4865 cpu_manager.go:414] "containerMap: removing container" podUID="e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" containerName="revoke-keys" Oct 31 16:25:38.555871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.555854 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" containerName="revoke-keys" Oct 31 16:25:38.556203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.556186 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" containerName="revoke-keys" Oct 31 16:25:38.560165 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.560144 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.612956 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.612771 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:25:38.612956 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.612897 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:25:38.623440 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.623396 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw"] Oct 31 16:25:38.657704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.657644 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:25:38.657940 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.657663 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:25:38.711982 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.711927 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.712216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.712095 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.712216 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.712144 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbz4h\" (UniqueName: \"kubernetes.io/projected/2233fabd-8b91-4137-86e7-08d5ac7ca568-kube-api-access-vbz4h\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.815118 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.815011 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbz4h\" (UniqueName: \"kubernetes.io/projected/2233fabd-8b91-4137-86e7-08d5ac7ca568-kube-api-access-vbz4h\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.815536 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.815506 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.815748 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.815727 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.816756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.816730 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.834928 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.834867 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.874698 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.874618 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbz4h\" (UniqueName: \"kubernetes.io/projected/2233fabd-8b91-4137-86e7-08d5ac7ca568-kube-api-access-vbz4h\") pod \"maas-api-revoke-sub-a4aef7d7-cpxqw\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:38.880861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:38.880633 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:39.840079 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:39.840049 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw"] Oct 31 16:25:40.150494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:40.128081 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" event={"ID":"2233fabd-8b91-4137-86e7-08d5ac7ca568","Type":"ContainerStarted","Data":"cb1dad0c4811ddd4d0ee506900f9da785e407c63f510dfbde03bcc2314150a61"} Oct 31 16:25:41.178695 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:41.178658 4865 generic.go:356] "Generic (PLEG): container finished" podID="2233fabd-8b91-4137-86e7-08d5ac7ca568" containerID="00f480bc0db9cbe62b1c216114b9e5e663a53600b9fa29518bef71facb622aa4" exitCode=0 Oct 31 16:25:41.179718 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:41.179317 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" event={"ID":"2233fabd-8b91-4137-86e7-08d5ac7ca568","Type":"ContainerDied","Data":"00f480bc0db9cbe62b1c216114b9e5e663a53600b9fa29518bef71facb622aa4"} Oct 31 16:25:41.651041 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:41.650557 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:25:41.651041 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:41.650623 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:25:42.018101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.018067 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:25:42.320617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.320593 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:42.433262 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.433228 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-xks-ca\") pod \"2233fabd-8b91-4137-86e7-08d5ac7ca568\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " Oct 31 16:25:42.433461 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.433313 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-openshift-ca\") pod \"2233fabd-8b91-4137-86e7-08d5ac7ca568\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " Oct 31 16:25:42.433461 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.433384 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2233fabd-8b91-4137-86e7-08d5ac7ca568-kube-api-access-vbz4h\" (UniqueName: \"kubernetes.io/projected/2233fabd-8b91-4137-86e7-08d5ac7ca568-kube-api-access-vbz4h\") pod \"2233fabd-8b91-4137-86e7-08d5ac7ca568\" (UID: \"2233fabd-8b91-4137-86e7-08d5ac7ca568\") " Oct 31 16:25:42.433768 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.433738 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-openshift-ca" pod "2233fabd-8b91-4137-86e7-08d5ac7ca568" (UID: "2233fabd-8b91-4137-86e7-08d5ac7ca568"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:25:42.436264 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.436151 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2233fabd-8b91-4137-86e7-08d5ac7ca568-kube-api-access-vbz4h" pod "2233fabd-8b91-4137-86e7-08d5ac7ca568" (UID: "2233fabd-8b91-4137-86e7-08d5ac7ca568"). InnerVolumeSpecName "kube-api-access-vbz4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:25:42.436547 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.436387 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-xks-ca" pod "2233fabd-8b91-4137-86e7-08d5ac7ca568" (UID: "2233fabd-8b91-4137-86e7-08d5ac7ca568"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:25:42.534794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.534762 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:25:42.534794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.534791 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2233fabd-8b91-4137-86e7-08d5ac7ca568-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:25:42.534962 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:42.534801 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbz4h\" (UniqueName: \"kubernetes.io/projected/2233fabd-8b91-4137-86e7-08d5ac7ca568-kube-api-access-vbz4h\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:25:43.195116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:43.195088 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" Oct 31 16:25:43.195116 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:43.195098 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw" event={"ID":"2233fabd-8b91-4137-86e7-08d5ac7ca568","Type":"ContainerDied","Data":"cb1dad0c4811ddd4d0ee506900f9da785e407c63f510dfbde03bcc2314150a61"} Oct 31 16:25:43.195356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:43.195151 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1dad0c4811ddd4d0ee506900f9da785e407c63f510dfbde03bcc2314150a61" Oct 31 16:25:47.490028 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:25:47.489987 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:25:50.309823 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:50.309781 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:25:56.477921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:56.477894 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:25:56.486419 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:56.486396 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:25:58.862363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:58.862332 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:25:58.870538 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:58.870519 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:25:59.099762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.099733 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:25:59.704966 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.704927 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl"] Oct 31 16:25:59.705890 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.705855 4865 cpu_manager.go:414] "containerMap: removing container" podUID="2233fabd-8b91-4137-86e7-08d5ac7ca568" containerName="revoke-keys" Oct 31 16:25:59.705890 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.705888 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2233fabd-8b91-4137-86e7-08d5ac7ca568" containerName="revoke-keys" Oct 31 16:25:59.706132 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.706037 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="2233fabd-8b91-4137-86e7-08d5ac7ca568" containerName="revoke-keys" Oct 31 16:25:59.709901 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.709874 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:25:59.770643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.770606 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl"] Oct 31 16:25:59.809798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.809510 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:25:59.809798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.809644 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:25:59.809798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.809752 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:25:59.815381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.815357 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:25:59.904649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.904598 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/a4fa45d8-a237-44d6-869d-4b4e35990299-kube-api-access-5j88k\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:25:59.905192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.904806 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:25:59.905192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:25:59.904886 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.006091 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.006045 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.006334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.006131 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.006334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.006185 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/a4fa45d8-a237-44d6-869d-4b4e35990299-kube-api-access-5j88k\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.006996 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.006969 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.009844 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.009814 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.063939 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.063900 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/a4fa45d8-a237-44d6-869d-4b4e35990299-kube-api-access-5j88k\") pod \"maas-api-revoke-sub-79805ca9-4n4kl\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.329875 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.329768 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:00.736778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.736738 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl"] Oct 31 16:26:00.754258 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:26:00.754199 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fa45d8_a237_44d6_869d_4b4e35990299.slice/crio-f7edaa4b6d853398e439b0cc9f0bee50d030d1f91c0b36e310d3cb76a338a8c1 WatchSource:0}: Error finding container f7edaa4b6d853398e439b0cc9f0bee50d030d1f91c0b36e310d3cb76a338a8c1: Status 404 returned error can't find the container with id f7edaa4b6d853398e439b0cc9f0bee50d030d1f91c0b36e310d3cb76a338a8c1 Oct 31 16:26:00.905502 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:00.896775 4865 scope.go:122] "RemoveContainer" containerID="6f5c41ccd6be6c9667847087376a438b36ad9014e15efdf2128c35f7fd8cccab" Oct 31 16:26:01.060490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:01.058708 4865 scope.go:122] "RemoveContainer" containerID="de3da22f915180c063ce3f22d8dcde11c953a1c878a5ee7c2d849d767e847b59" Oct 31 16:26:01.362493 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:01.360398 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" event={"ID":"a4fa45d8-a237-44d6-869d-4b4e35990299","Type":"ContainerStarted","Data":"f7edaa4b6d853398e439b0cc9f0bee50d030d1f91c0b36e310d3cb76a338a8c1"} Oct 31 16:26:02.400506 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:02.400444 4865 generic.go:356] "Generic (PLEG): container finished" podID="a4fa45d8-a237-44d6-869d-4b4e35990299" containerID="6bbfc132f72d876d09c4b021e387998522bce0d6ad24a069ce14e56a100acb5e" exitCode=0 Oct 31 16:26:02.401146 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:02.401119 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" event={"ID":"a4fa45d8-a237-44d6-869d-4b4e35990299","Type":"ContainerDied","Data":"6bbfc132f72d876d09c4b021e387998522bce0d6ad24a069ce14e56a100acb5e"} Oct 31 16:26:03.665439 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.665414 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:03.751766 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.751738 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fa45d8-a237-44d6-869d-4b4e35990299-kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/a4fa45d8-a237-44d6-869d-4b4e35990299-kube-api-access-5j88k\") pod \"a4fa45d8-a237-44d6-869d-4b4e35990299\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " Oct 31 16:26:03.751948 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.751810 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-openshift-ca\") pod \"a4fa45d8-a237-44d6-869d-4b4e35990299\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " Oct 31 16:26:03.751948 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.751893 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-xks-ca\") pod \"a4fa45d8-a237-44d6-869d-4b4e35990299\" (UID: \"a4fa45d8-a237-44d6-869d-4b4e35990299\") " Oct 31 16:26:03.752172 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.752149 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-openshift-ca" pod "a4fa45d8-a237-44d6-869d-4b4e35990299" (UID: "a4fa45d8-a237-44d6-869d-4b4e35990299"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:03.754072 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.754051 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fa45d8-a237-44d6-869d-4b4e35990299-kube-api-access-5j88k" pod "a4fa45d8-a237-44d6-869d-4b4e35990299" (UID: "a4fa45d8-a237-44d6-869d-4b4e35990299"). InnerVolumeSpecName "kube-api-access-5j88k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:03.754072 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.754062 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-xks-ca" pod "a4fa45d8-a237-44d6-869d-4b4e35990299" (UID: "a4fa45d8-a237-44d6-869d-4b4e35990299"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:03.852959 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.852925 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:03.852959 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.852957 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j88k\" (UniqueName: \"kubernetes.io/projected/a4fa45d8-a237-44d6-869d-4b4e35990299-kube-api-access-5j88k\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:03.853163 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:03.852967 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fa45d8-a237-44d6-869d-4b4e35990299-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:04.425322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:04.425286 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" event={"ID":"a4fa45d8-a237-44d6-869d-4b4e35990299","Type":"ContainerDied","Data":"f7edaa4b6d853398e439b0cc9f0bee50d030d1f91c0b36e310d3cb76a338a8c1"} Oct 31 16:26:04.425322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:04.425317 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl" Oct 31 16:26:04.425575 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:04.425363 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7edaa4b6d853398e439b0cc9f0bee50d030d1f91c0b36e310d3cb76a338a8c1" Oct 31 16:26:08.307238 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:08.307208 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:26:12.264484 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:12.264444 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:17.161496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.161301 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm"] Oct 31 16:26:17.162498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.162085 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a4fa45d8-a237-44d6-869d-4b4e35990299" containerName="revoke-keys" Oct 31 16:26:17.162498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.162121 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fa45d8-a237-44d6-869d-4b4e35990299" containerName="revoke-keys" Oct 31 16:26:17.162498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.162302 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fa45d8-a237-44d6-869d-4b4e35990299" containerName="revoke-keys" Oct 31 16:26:17.176272 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.174741 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.219669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.219638 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm"] Oct 31 16:26:17.258643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.258391 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:26:17.258643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.258521 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:26:17.258643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.258600 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:26:17.348643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.348608 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:26:17.370043 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.370004 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.372301 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.372276 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkbx\" (UniqueName: \"kubernetes.io/projected/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-kube-api-access-qxkbx\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.372628 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.372608 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.475159 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.475106 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.475490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.475447 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkbx\" (UniqueName: \"kubernetes.io/projected/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-kube-api-access-qxkbx\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.475742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.475722 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.479584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.479546 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.527710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.527678 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.645793 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:26:17.645754 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:26:17.658495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.655151 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkbx\" (UniqueName: \"kubernetes.io/projected/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-kube-api-access-qxkbx\") pod \"maas-api-revoke-sub-0b7e8edb-mqtwm\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:17.804548 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:17.803600 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:18.794704 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:18.794669 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm"] Oct 31 16:26:19.595817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:19.595777 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:19.799239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:19.799210 4865 generic.go:356] "Generic (PLEG): container finished" podID="8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" containerID="e00b0249f64bdc469ee73d04088cf0437b6e78284378b72845a99386e6852344" exitCode=0 Oct 31 16:26:19.799681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:19.799287 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" event={"ID":"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265","Type":"ContainerDied","Data":"e00b0249f64bdc469ee73d04088cf0437b6e78284378b72845a99386e6852344"} Oct 31 16:26:19.799681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:19.799346 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" event={"ID":"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265","Type":"ContainerStarted","Data":"6797f0ecddfa4d336ef815fde8c4b3cc3908173d4845627993ca47554595a522"} Oct 31 16:26:20.711898 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:20.711866 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:26:20.952219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:20.952197 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:21.020381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.020283 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-openshift-ca\") pod \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " Oct 31 16:26:21.020381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.020331 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-kube-api-access-qxkbx\" (UniqueName: \"kubernetes.io/projected/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-kube-api-access-qxkbx\") pod \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " Oct 31 16:26:21.020699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.020457 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-xks-ca\") pod \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\" (UID: \"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265\") " Oct 31 16:26:21.020699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.020687 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-openshift-ca" pod "8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" (UID: "8beee6d1-08f8-48b8-9a1d-4ab94ea0a265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:21.020832 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.020811 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:21.022774 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.022746 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-kube-api-access-qxkbx" pod "8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" (UID: "8beee6d1-08f8-48b8-9a1d-4ab94ea0a265"). InnerVolumeSpecName "kube-api-access-qxkbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:21.022905 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.022830 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-xks-ca" pod "8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" (UID: "8beee6d1-08f8-48b8-9a1d-4ab94ea0a265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:21.087938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.087902 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:21.121924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.121897 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkbx\" (UniqueName: \"kubernetes.io/projected/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-kube-api-access-qxkbx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:21.121924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.121925 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:21.810671 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.810637 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" Oct 31 16:26:21.810869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.810671 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm" event={"ID":"8beee6d1-08f8-48b8-9a1d-4ab94ea0a265","Type":"ContainerDied","Data":"6797f0ecddfa4d336ef815fde8c4b3cc3908173d4845627993ca47554595a522"} Oct 31 16:26:21.810869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:21.810722 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6797f0ecddfa4d336ef815fde8c4b3cc3908173d4845627993ca47554595a522" Oct 31 16:26:22.470365 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:22.470326 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:28.468308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.468272 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd"] Oct 31 16:26:28.468817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.468715 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" containerName="revoke-keys" Oct 31 16:26:28.468817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.468727 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" containerName="revoke-keys" Oct 31 16:26:28.468817 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.468808 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" containerName="revoke-keys" Oct 31 16:26:28.471748 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.471731 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.482606 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.482581 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zh88\" (UniqueName: \"kubernetes.io/projected/06e0ba40-3e1f-413f-86c6-553596260a82-kube-api-access-8zh88\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.482738 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.482714 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.482813 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.482760 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.527869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.527839 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd"] Oct 31 16:26:28.573087 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.573061 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:26:28.573306 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.573288 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:26:28.573794 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.573777 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:26:28.573988 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.573969 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:26:28.583461 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.583426 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.583587 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.583497 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.583587 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.583540 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zh88\" (UniqueName: \"kubernetes.io/projected/06e0ba40-3e1f-413f-86c6-553596260a82-kube-api-access-8zh88\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.584156 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.584134 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.586002 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.585982 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.642946 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.642919 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zh88\" (UniqueName: \"kubernetes.io/projected/06e0ba40-3e1f-413f-86c6-553596260a82-kube-api-access-8zh88\") pod \"maas-api-revoke-sub-3d4156f1-wslmd\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.785743 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.785657 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:28.938664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:28.938633 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:26:29.157874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:29.157844 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd"] Oct 31 16:26:29.158392 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:26:29.158362 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e0ba40_3e1f_413f_86c6_553596260a82.slice/crio-a7ec6f6e5bd58a3cf79b909d0a1e8e6c9c9842855f5d9705c0b1e2ccf1e470c7 WatchSource:0}: Error finding container a7ec6f6e5bd58a3cf79b909d0a1e8e6c9c9842855f5d9705c0b1e2ccf1e470c7: Status 404 returned error can't find the container with id a7ec6f6e5bd58a3cf79b909d0a1e8e6c9c9842855f5d9705c0b1e2ccf1e470c7 Oct 31 16:26:29.868250 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:29.868212 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" event={"ID":"06e0ba40-3e1f-413f-86c6-553596260a82","Type":"ContainerStarted","Data":"241f24ecd4e22fb51d83e30cca0461137e71e58a40e0e90c043bbc5f7a04a724"} Oct 31 16:26:29.868755 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:29.868272 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" event={"ID":"06e0ba40-3e1f-413f-86c6-553596260a82","Type":"ContainerStarted","Data":"a7ec6f6e5bd58a3cf79b909d0a1e8e6c9c9842855f5d9705c0b1e2ccf1e470c7"} Oct 31 16:26:29.982225 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:29.982169 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" podStartSLOduration=1.9821538410000001 podStartE2EDuration="1.982153841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:29.979381291 +0000 UTC m=+6035.598932703" watchObservedRunningTime="2026-10-31 16:26:29.982153841 +0000 UTC m=+6035.601705251" Oct 31 16:26:30.875020 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:30.874984 4865 generic.go:356] "Generic (PLEG): container finished" podID="06e0ba40-3e1f-413f-86c6-553596260a82" containerID="241f24ecd4e22fb51d83e30cca0461137e71e58a40e0e90c043bbc5f7a04a724" exitCode=0 Oct 31 16:26:30.875457 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:30.875075 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" event={"ID":"06e0ba40-3e1f-413f-86c6-553596260a82","Type":"ContainerDied","Data":"241f24ecd4e22fb51d83e30cca0461137e71e58a40e0e90c043bbc5f7a04a724"} Oct 31 16:26:32.015362 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.015337 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:32.117912 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.117877 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-openshift-ca\") pod \"06e0ba40-3e1f-413f-86c6-553596260a82\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " Oct 31 16:26:32.118106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.117947 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-xks-ca\") pod \"06e0ba40-3e1f-413f-86c6-553596260a82\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " Oct 31 16:26:32.118106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.117974 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e0ba40-3e1f-413f-86c6-553596260a82-kube-api-access-8zh88\" (UniqueName: \"kubernetes.io/projected/06e0ba40-3e1f-413f-86c6-553596260a82-kube-api-access-8zh88\") pod \"06e0ba40-3e1f-413f-86c6-553596260a82\" (UID: \"06e0ba40-3e1f-413f-86c6-553596260a82\") " Oct 31 16:26:32.118280 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.118256 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-openshift-ca" pod "06e0ba40-3e1f-413f-86c6-553596260a82" (UID: "06e0ba40-3e1f-413f-86c6-553596260a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:32.120220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.120197 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-xks-ca" pod "06e0ba40-3e1f-413f-86c6-553596260a82" (UID: "06e0ba40-3e1f-413f-86c6-553596260a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:32.120220 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.120197 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e0ba40-3e1f-413f-86c6-553596260a82-kube-api-access-8zh88" pod "06e0ba40-3e1f-413f-86c6-553596260a82" (UID: "06e0ba40-3e1f-413f-86c6-553596260a82"). InnerVolumeSpecName "kube-api-access-8zh88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:32.219513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.219463 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:32.219513 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.219513 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06e0ba40-3e1f-413f-86c6-553596260a82-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:32.219733 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.219527 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zh88\" (UniqueName: \"kubernetes.io/projected/06e0ba40-3e1f-413f-86c6-553596260a82-kube-api-access-8zh88\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:32.886756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.886671 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" Oct 31 16:26:32.886756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.886682 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd" event={"ID":"06e0ba40-3e1f-413f-86c6-553596260a82","Type":"ContainerDied","Data":"a7ec6f6e5bd58a3cf79b909d0a1e8e6c9c9842855f5d9705c0b1e2ccf1e470c7"} Oct 31 16:26:32.886756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:32.886733 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7ec6f6e5bd58a3cf79b909d0a1e8e6c9c9842855f5d9705c0b1e2ccf1e470c7" Oct 31 16:26:43.094886 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:43.094846 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:26:47.490143 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:26:47.490103 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:26:47.866156 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:47.866123 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd"] Oct 31 16:26:47.916076 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:47.916043 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wslmd"] Oct 31 16:26:47.985924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:47.985892 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92"] Oct 31 16:26:47.986317 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:47.986304 4865 cpu_manager.go:414] "containerMap: removing container" podUID="06e0ba40-3e1f-413f-86c6-553596260a82" containerName="revoke-keys" Oct 31 16:26:47.986379 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:47.986319 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e0ba40-3e1f-413f-86c6-553596260a82" containerName="revoke-keys" Oct 31 16:26:47.986420 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:47.986412 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e0ba40-3e1f-413f-86c6-553596260a82" containerName="revoke-keys" Oct 31 16:26:47.989549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:47.989531 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.042413 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.042387 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:26:48.042614 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.042494 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:26:48.042614 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.042520 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:26:48.042614 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.042533 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:26:48.045335 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.045314 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92"] Oct 31 16:26:48.144446 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.144367 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.144446 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.144414 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.144679 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.144504 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxgbg\" (UniqueName: \"kubernetes.io/projected/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-kube-api-access-vxgbg\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.245184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.245150 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.245355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.245193 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.245355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.245210 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxgbg\" (UniqueName: \"kubernetes.io/projected/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-kube-api-access-vxgbg\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.245851 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.245820 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.247883 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.247863 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.301819 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.301790 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxgbg\" (UniqueName: \"kubernetes.io/projected/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-kube-api-access-vxgbg\") pod \"maas-api-revoke-sub-3d4156f1-5ds92\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.302499 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.302463 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:48.495407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.495374 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e0ba40-3e1f-413f-86c6-553596260a82" path="/var/lib/kubelet/pods/06e0ba40-3e1f-413f-86c6-553596260a82/volumes" Oct 31 16:26:48.675140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:48.675112 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92"] Oct 31 16:26:48.676440 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:26:48.676402 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe4a4b2_ffb7_46bb_93fb_443615d0eaf5.slice/crio-5c93878567da53f6eda83f47a09c2589f1dbe5b7059921c9d674516292e3a7fc WatchSource:0}: Error finding container 5c93878567da53f6eda83f47a09c2589f1dbe5b7059921c9d674516292e3a7fc: Status 404 returned error can't find the container with id 5c93878567da53f6eda83f47a09c2589f1dbe5b7059921c9d674516292e3a7fc Oct 31 16:26:49.000895 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:49.000859 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" event={"ID":"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5","Type":"ContainerStarted","Data":"5c93878567da53f6eda83f47a09c2589f1dbe5b7059921c9d674516292e3a7fc"} Oct 31 16:26:50.008595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:50.008559 4865 generic.go:356] "Generic (PLEG): container finished" podID="4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" containerID="9373deaded8d1f439c4199480bd2d557ade09ce3545a8a97cd5597bd40a65a6c" exitCode=0 Oct 31 16:26:50.009025 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:50.008637 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" event={"ID":"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5","Type":"ContainerDied","Data":"9373deaded8d1f439c4199480bd2d557ade09ce3545a8a97cd5597bd40a65a6c"} Oct 31 16:26:51.149996 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.149969 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:51.273140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.273044 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-openshift-ca\") pod \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " Oct 31 16:26:51.273330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.273147 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-xks-ca\") pod \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " Oct 31 16:26:51.273330 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.273215 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-kube-api-access-vxgbg\" (UniqueName: \"kubernetes.io/projected/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-kube-api-access-vxgbg\") pod \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\" (UID: \"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5\") " Oct 31 16:26:51.273493 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.273433 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-openshift-ca" pod "4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" (UID: "4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:51.273612 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.273588 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:51.275929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.275906 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-xks-ca" pod "4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" (UID: "4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:51.276039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.275931 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-kube-api-access-vxgbg" pod "4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" (UID: "4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5"). InnerVolumeSpecName "kube-api-access-vxgbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:51.375153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.375121 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:51.375153 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:51.375152 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxgbg\" (UniqueName: \"kubernetes.io/projected/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5-kube-api-access-vxgbg\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:26:52.022855 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:52.022823 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" Oct 31 16:26:52.022855 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:52.022832 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92" event={"ID":"4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5","Type":"ContainerDied","Data":"5c93878567da53f6eda83f47a09c2589f1dbe5b7059921c9d674516292e3a7fc"} Oct 31 16:26:52.023085 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:26:52.022884 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c93878567da53f6eda83f47a09c2589f1dbe5b7059921c9d674516292e3a7fc" Oct 31 16:27:01.288019 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:01.287984 4865 scope.go:122] "RemoveContainer" containerID="69b9e67758aec9490a9cccbf844105067ecb64a20419d040a1d36b17d2fdf430" Oct 31 16:27:13.901344 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:13.901261 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:27:17.620496 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:27:17.617662 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:27:20.025524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:20.025459 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:21.426047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:21.426014 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:22.561898 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.561856 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk"] Oct 31 16:27:22.562325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.562299 4865 cpu_manager.go:414] "containerMap: removing container" podUID="4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" containerName="revoke-keys" Oct 31 16:27:22.562325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.562311 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" containerName="revoke-keys" Oct 31 16:27:22.562418 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.562388 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" containerName="revoke-keys" Oct 31 16:27:22.565509 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.565490 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.621753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.621722 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk"] Oct 31 16:27:22.666046 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.666013 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:27:22.666046 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.666024 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:27:22.666310 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.666084 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:27:22.667554 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.667538 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:27:22.694175 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.694147 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.694319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.694190 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.694319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.694207 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rqkx\" (UniqueName: \"kubernetes.io/projected/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-kube-api-access-5rqkx\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.795121 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.795090 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.795295 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.795133 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.795295 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.795255 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rqkx\" (UniqueName: \"kubernetes.io/projected/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-kube-api-access-5rqkx\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.795768 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.795746 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.797545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.797523 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.816205 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.816153 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:22.851639 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.851611 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rqkx\" (UniqueName: \"kubernetes.io/projected/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-kube-api-access-5rqkx\") pod \"maas-api-revoke-sub-e2fc9c5b-5qfsk\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:22.878855 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:22.878832 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:23.248370 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:23.248330 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk"] Oct 31 16:27:23.248744 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:27:23.248718 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9503c8_0823_4bf9_a1e8_f070d56b4e79.slice/crio-7ac29b5569c6f7c6ba080d6399ea1bb6922984023bd2860e169728590e2c3c1a WatchSource:0}: Error finding container 7ac29b5569c6f7c6ba080d6399ea1bb6922984023bd2860e169728590e2c3c1a: Status 404 returned error can't find the container with id 7ac29b5569c6f7c6ba080d6399ea1bb6922984023bd2860e169728590e2c3c1a Oct 31 16:27:24.219752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:24.219712 4865 generic.go:356] "Generic (PLEG): container finished" podID="6e9503c8-0823-4bf9-a1e8-f070d56b4e79" containerID="b5f7b39388185f5f0f1db239aab18e7be6bb343f14caebacf8fa41a7d851f156" exitCode=0 Oct 31 16:27:24.220162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:24.219789 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" event={"ID":"6e9503c8-0823-4bf9-a1e8-f070d56b4e79","Type":"ContainerDied","Data":"b5f7b39388185f5f0f1db239aab18e7be6bb343f14caebacf8fa41a7d851f156"} Oct 31 16:27:24.220162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:24.219840 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" event={"ID":"6e9503c8-0823-4bf9-a1e8-f070d56b4e79","Type":"ContainerStarted","Data":"7ac29b5569c6f7c6ba080d6399ea1bb6922984023bd2860e169728590e2c3c1a"} Oct 31 16:27:25.361806 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.361777 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:25.521144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.521070 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-openshift-ca\") pod \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " Oct 31 16:27:25.521144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.521137 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-kube-api-access-5rqkx\" (UniqueName: \"kubernetes.io/projected/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-kube-api-access-5rqkx\") pod \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " Oct 31 16:27:25.521347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.521267 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-xks-ca\") pod \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\" (UID: \"6e9503c8-0823-4bf9-a1e8-f070d56b4e79\") " Oct 31 16:27:25.521521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.521497 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-openshift-ca" pod "6e9503c8-0823-4bf9-a1e8-f070d56b4e79" (UID: "6e9503c8-0823-4bf9-a1e8-f070d56b4e79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:27:25.521645 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.521621 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:25.523346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.523323 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-xks-ca" pod "6e9503c8-0823-4bf9-a1e8-f070d56b4e79" (UID: "6e9503c8-0823-4bf9-a1e8-f070d56b4e79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:27:25.523490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.523456 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-kube-api-access-5rqkx" pod "6e9503c8-0823-4bf9-a1e8-f070d56b4e79" (UID: "6e9503c8-0823-4bf9-a1e8-f070d56b4e79"). InnerVolumeSpecName "kube-api-access-5rqkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:27:25.622887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.622861 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rqkx\" (UniqueName: \"kubernetes.io/projected/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-kube-api-access-5rqkx\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:25.622887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:25.622886 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9503c8-0823-4bf9-a1e8-f070d56b4e79-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:26.232035 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:26.232004 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" event={"ID":"6e9503c8-0823-4bf9-a1e8-f070d56b4e79","Type":"ContainerDied","Data":"7ac29b5569c6f7c6ba080d6399ea1bb6922984023bd2860e169728590e2c3c1a"} Oct 31 16:27:26.232035 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:26.232027 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk" Oct 31 16:27:26.232275 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:26.232053 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ac29b5569c6f7c6ba080d6399ea1bb6922984023bd2860e169728590e2c3c1a" Oct 31 16:27:33.362783 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.362748 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn"] Oct 31 16:27:33.363323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.363186 4865 cpu_manager.go:414] "containerMap: removing container" podUID="6e9503c8-0823-4bf9-a1e8-f070d56b4e79" containerName="revoke-keys" Oct 31 16:27:33.363323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.363198 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9503c8-0823-4bf9-a1e8-f070d56b4e79" containerName="revoke-keys" Oct 31 16:27:33.363323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.363283 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9503c8-0823-4bf9-a1e8-f070d56b4e79" containerName="revoke-keys" Oct 31 16:27:33.366383 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.366366 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.395019 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.394990 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hv4\" (UniqueName: \"kubernetes.io/projected/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-kube-api-access-49hv4\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.395182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.395051 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.395182 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.395171 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.424027 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.423993 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn"] Oct 31 16:27:33.467142 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.467101 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:27:33.467357 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.467332 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:27:33.467452 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.467384 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:27:33.468235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.468216 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:27:33.495862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.495837 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.495978 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.495888 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.495978 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.495930 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hv4\" (UniqueName: \"kubernetes.io/projected/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-kube-api-access-49hv4\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.496528 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.496506 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.498355 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.498333 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.552208 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.552178 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hv4\" (UniqueName: \"kubernetes.io/projected/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-kube-api-access-49hv4\") pod \"maas-api-revoke-sub-5a29efb3-mzmmn\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:33.686344 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:33.686268 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:34.058813 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:34.058790 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn"] Oct 31 16:27:34.059725 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:27:34.059693 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949adb0d_c6d2_48d6_a8ea_aa8be62b184e.slice/crio-3bac364ef06e5404f6bf205b32e038d03b15934a32b50c5d1424421c7f030659 WatchSource:0}: Error finding container 3bac364ef06e5404f6bf205b32e038d03b15934a32b50c5d1424421c7f030659: Status 404 returned error can't find the container with id 3bac364ef06e5404f6bf205b32e038d03b15934a32b50c5d1424421c7f030659 Oct 31 16:27:34.272930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:34.272894 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" event={"ID":"949adb0d-c6d2-48d6-a8ea-aa8be62b184e","Type":"ContainerStarted","Data":"3bac364ef06e5404f6bf205b32e038d03b15934a32b50c5d1424421c7f030659"} Oct 31 16:27:35.279441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:35.279404 4865 generic.go:356] "Generic (PLEG): container finished" podID="949adb0d-c6d2-48d6-a8ea-aa8be62b184e" containerID="cc16386a104b7d46d6e25c567709d1c8c4509d3c3739f421e76b63b18768650e" exitCode=0 Oct 31 16:27:35.279881 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:35.279497 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" event={"ID":"949adb0d-c6d2-48d6-a8ea-aa8be62b184e","Type":"ContainerDied","Data":"cc16386a104b7d46d6e25c567709d1c8c4509d3c3739f421e76b63b18768650e"} Oct 31 16:27:35.437112 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:35.437070 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:27:36.484000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.483964 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:36.525105 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.525065 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-openshift-ca\") pod \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " Oct 31 16:27:36.525316 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.525254 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-xks-ca\") pod \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " Oct 31 16:27:36.525316 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.525293 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-kube-api-access-49hv4\" (UniqueName: \"kubernetes.io/projected/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-kube-api-access-49hv4\") pod \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\" (UID: \"949adb0d-c6d2-48d6-a8ea-aa8be62b184e\") " Oct 31 16:27:36.525563 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.525534 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-openshift-ca" pod "949adb0d-c6d2-48d6-a8ea-aa8be62b184e" (UID: "949adb0d-c6d2-48d6-a8ea-aa8be62b184e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:27:36.525687 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.525648 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:36.528245 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.528213 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-kube-api-access-49hv4" pod "949adb0d-c6d2-48d6-a8ea-aa8be62b184e" (UID: "949adb0d-c6d2-48d6-a8ea-aa8be62b184e"). InnerVolumeSpecName "kube-api-access-49hv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:27:36.528353 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.528297 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-xks-ca" pod "949adb0d-c6d2-48d6-a8ea-aa8be62b184e" (UID: "949adb0d-c6d2-48d6-a8ea-aa8be62b184e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:27:36.627318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.627217 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:36.627318 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:36.627262 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49hv4\" (UniqueName: \"kubernetes.io/projected/949adb0d-c6d2-48d6-a8ea-aa8be62b184e-kube-api-access-49hv4\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:37.322494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:37.317259 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" event={"ID":"949adb0d-c6d2-48d6-a8ea-aa8be62b184e","Type":"ContainerDied","Data":"3bac364ef06e5404f6bf205b32e038d03b15934a32b50c5d1424421c7f030659"} Oct 31 16:27:37.322494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:37.317325 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bac364ef06e5404f6bf205b32e038d03b15934a32b50c5d1424421c7f030659" Oct 31 16:27:37.322494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:37.317398 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn" Oct 31 16:27:38.317307 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:38.310820 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:27:38.317307 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:38.310893 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:27:38.694571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:38.692569 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:27:38.694571 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:38.692689 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:27:46.090598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:46.090559 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:27:47.489255 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:27:47.489206 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:27:50.382459 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.382395 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92"] Oct 31 16:27:50.433965 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.433901 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5ds92"] Oct 31 16:27:50.495006 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.494969 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5" path="/var/lib/kubelet/pods/4fe4a4b2-ffb7-46bb-93fb-443615d0eaf5/volumes" Oct 31 16:27:50.506688 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.506652 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj"] Oct 31 16:27:50.507268 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.507251 4865 cpu_manager.go:414] "containerMap: removing container" podUID="949adb0d-c6d2-48d6-a8ea-aa8be62b184e" containerName="revoke-keys" Oct 31 16:27:50.507321 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.507272 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949adb0d-c6d2-48d6-a8ea-aa8be62b184e" containerName="revoke-keys" Oct 31 16:27:50.507398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.507386 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="949adb0d-c6d2-48d6-a8ea-aa8be62b184e" containerName="revoke-keys" Oct 31 16:27:50.511211 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.511185 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.563689 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.563654 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:27:50.563870 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.563851 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:27:50.563938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.563912 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:27:50.564592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.564571 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:27:50.568431 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.568406 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj"] Oct 31 16:27:50.647771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.647667 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.647952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.647813 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.647952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.647933 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz57c\" (UniqueName: \"kubernetes.io/projected/79fb1b70-1d98-41a8-8896-73b931915953-kube-api-access-fz57c\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.749244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.749199 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz57c\" (UniqueName: \"kubernetes.io/projected/79fb1b70-1d98-41a8-8896-73b931915953-kube-api-access-fz57c\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.749483 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.749281 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.749483 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.749346 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.750043 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.750015 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.751854 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.751834 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.805623 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.805589 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz57c\" (UniqueName: \"kubernetes.io/projected/79fb1b70-1d98-41a8-8896-73b931915953-kube-api-access-fz57c\") pod \"maas-api-revoke-sub-3d4156f1-mh8pj\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:50.825390 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:50.825361 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:51.198512 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:51.198462 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj"] Oct 31 16:27:51.198924 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:27:51.198898 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fb1b70_1d98_41a8_8896_73b931915953.slice/crio-afd03bd3dd045707db3cad29752ab9e3045a8add4d0af7c476977e8f2515a90d WatchSource:0}: Error finding container afd03bd3dd045707db3cad29752ab9e3045a8add4d0af7c476977e8f2515a90d: Status 404 returned error can't find the container with id afd03bd3dd045707db3cad29752ab9e3045a8add4d0af7c476977e8f2515a90d Oct 31 16:27:51.426847 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:51.426805 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" event={"ID":"79fb1b70-1d98-41a8-8896-73b931915953","Type":"ContainerStarted","Data":"afd03bd3dd045707db3cad29752ab9e3045a8add4d0af7c476977e8f2515a90d"} Oct 31 16:27:52.437641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:52.437546 4865 generic.go:356] "Generic (PLEG): container finished" podID="79fb1b70-1d98-41a8-8896-73b931915953" containerID="09fd8a4cd3d9520dac32431eab587ce1a18331a3e95b082a3f09e1adbb378859" exitCode=0 Oct 31 16:27:52.437641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:52.437603 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" event={"ID":"79fb1b70-1d98-41a8-8896-73b931915953","Type":"ContainerDied","Data":"09fd8a4cd3d9520dac32431eab587ce1a18331a3e95b082a3f09e1adbb378859"} Oct 31 16:27:53.579772 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.579743 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:53.677371 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.677334 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fb1b70-1d98-41a8-8896-73b931915953-kube-api-access-fz57c\" (UniqueName: \"kubernetes.io/projected/79fb1b70-1d98-41a8-8896-73b931915953-kube-api-access-fz57c\") pod \"79fb1b70-1d98-41a8-8896-73b931915953\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " Oct 31 16:27:53.677614 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.677444 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-xks-ca\") pod \"79fb1b70-1d98-41a8-8896-73b931915953\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " Oct 31 16:27:53.677614 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.677511 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-openshift-ca\") pod \"79fb1b70-1d98-41a8-8896-73b931915953\" (UID: \"79fb1b70-1d98-41a8-8896-73b931915953\") " Oct 31 16:27:53.677864 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.677835 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-openshift-ca" pod "79fb1b70-1d98-41a8-8896-73b931915953" (UID: "79fb1b70-1d98-41a8-8896-73b931915953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:27:53.679617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.679594 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fb1b70-1d98-41a8-8896-73b931915953-kube-api-access-fz57c" pod "79fb1b70-1d98-41a8-8896-73b931915953" (UID: "79fb1b70-1d98-41a8-8896-73b931915953"). InnerVolumeSpecName "kube-api-access-fz57c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:27:53.679712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.679616 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-xks-ca" pod "79fb1b70-1d98-41a8-8896-73b931915953" (UID: "79fb1b70-1d98-41a8-8896-73b931915953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:27:53.778505 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.778448 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz57c\" (UniqueName: \"kubernetes.io/projected/79fb1b70-1d98-41a8-8896-73b931915953-kube-api-access-fz57c\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:53.778505 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.778503 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:53.778750 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:53.778516 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79fb1b70-1d98-41a8-8896-73b931915953-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:27:54.449066 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:54.449031 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" Oct 31 16:27:54.449260 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:54.449032 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj" event={"ID":"79fb1b70-1d98-41a8-8896-73b931915953","Type":"ContainerDied","Data":"afd03bd3dd045707db3cad29752ab9e3045a8add4d0af7c476977e8f2515a90d"} Oct 31 16:27:54.449260 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:54.449156 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd03bd3dd045707db3cad29752ab9e3045a8add4d0af7c476977e8f2515a90d" Oct 31 16:27:55.044774 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:55.044735 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh"] Oct 31 16:27:55.094314 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:55.094275 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-7jbjh"] Oct 31 16:27:56.494636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:27:56.494603 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0" path="/var/lib/kubelet/pods/e4aaae81-2c7f-4f6c-8f54-f9fe39f772f0/volumes" Oct 31 16:28:06.658998 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:06.658963 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:07.188163 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.188127 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs"] Oct 31 16:28:07.188565 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.188550 4865 cpu_manager.go:414] "containerMap: removing container" podUID="79fb1b70-1d98-41a8-8896-73b931915953" containerName="revoke-keys" Oct 31 16:28:07.188628 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.188567 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fb1b70-1d98-41a8-8896-73b931915953" containerName="revoke-keys" Oct 31 16:28:07.188669 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.188640 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fb1b70-1d98-41a8-8896-73b931915953" containerName="revoke-keys" Oct 31 16:28:07.192934 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.192917 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.245125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.245097 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-bcf50e-openshift-default-dockercfg-62hv4\"" Oct 31 16:28:07.298325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.298295 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs"] Oct 31 16:28:07.300033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300007 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-envoy\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300162 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300057 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252hh\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-kube-api-access-252hh\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300211 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300169 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-token\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300266 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300211 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd65f1-4615-4104-a848-a23f7e681b79-istiod-ca-cert\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300328 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300312 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-data\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300396 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300346 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-podinfo\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300396 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300376 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-socket\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300424 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-credential-socket\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.300518 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.300512 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-certs\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.333269 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.333232 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:07.401541 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401502 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252hh\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-kube-api-access-252hh\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.401752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401559 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-token\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.401752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401593 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd65f1-4615-4104-a848-a23f7e681b79-istiod-ca-cert\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.401752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401628 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-data\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.401752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401647 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-podinfo\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.401752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401663 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-socket\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.401752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401688 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-credential-socket\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.401752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401718 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-certs\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.402188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.401805 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-envoy\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.402188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.402119 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-data\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.402188 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.402176 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-credential-socket\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.402393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.402259 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-socket\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.402393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.402308 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-certs\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.402550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.402441 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd65f1-4615-4104-a848-a23f7e681b79-istiod-ca-cert\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.404305 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.404287 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-envoy\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.404584 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.404566 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-podinfo\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.461619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.461596 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-token\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.461911 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.461889 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-252hh\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-kube-api-access-252hh\") pod \"e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.506047 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.506015 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:07.620690 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.620642 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:07.932210 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.932176 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs"] Oct 31 16:28:07.932637 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:28:07.932321 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbd65f1_4615_4104_a848_a23f7e681b79.slice/crio-41b4d2cc2a46bcf7948293dab3b944a21bbad21d2fd582c3e52cb95911d3f142 WatchSource:0}: Error finding container 41b4d2cc2a46bcf7948293dab3b944a21bbad21d2fd582c3e52cb95911d3f142: Status 404 returned error can't find the container with id 41b4d2cc2a46bcf7948293dab3b944a21bbad21d2fd582c3e52cb95911d3f142 Oct 31 16:28:07.934569 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.934540 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:28:07.934646 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.934599 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:28:07.934646 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:07.934626 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:28:08.520782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:08.520736 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" event={"ID":"fdbd65f1-4615-4104-a848-a23f7e681b79","Type":"ContainerStarted","Data":"ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f"} Oct 31 16:28:08.520969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:08.520807 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" event={"ID":"fdbd65f1-4615-4104-a848-a23f7e681b79","Type":"ContainerStarted","Data":"41b4d2cc2a46bcf7948293dab3b944a21bbad21d2fd582c3e52cb95911d3f142"} Oct 31 16:28:08.639966 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:08.639909 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" podStartSLOduration=1.639892978 podStartE2EDuration="1.639892978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:08.635191639 +0000 UTC m=+6134.254743030" watchObservedRunningTime="2026-10-31 16:28:08.639892978 +0000 UTC m=+6134.259444389" Oct 31 16:28:09.507215 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:09.507180 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:09.512331 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:09.512305 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:09.534752 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:09.534724 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:09.535573 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:09.535552 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:11.506622 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:11.506593 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:11.692930 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:11.692896 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:11.859684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:11.859608 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:14.751715 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:14.751683 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:14.921831 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:14.921800 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:15.167778 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.167697 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9"] Oct 31 16:28:15.171617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.171601 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.227516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.227483 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9"] Oct 31 16:28:15.272641 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.272607 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-bcf50e\"" Oct 31 16:28:15.273596 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.272693 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9fd1a5ac-dockercfg-82z87\"" Oct 31 16:28:15.283514 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.283438 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00feb578-83bd-4297-bf32-2fc074d5a617-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.283673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.283550 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj7db\" (UniqueName: \"kubernetes.io/projected/00feb578-83bd-4297-bf32-2fc074d5a617-kube-api-access-cj7db\") pod \"payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.325771 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.325746 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj"] Oct 31 16:28:15.329007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.328990 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.384302 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.384267 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj"] Oct 31 16:28:15.384763 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.384732 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031cc8de-5bb9-4f1a-a40c-220fac42948d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.384911 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.384891 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00feb578-83bd-4297-bf32-2fc074d5a617-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.384959 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.384938 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjmj8\" (UniqueName: \"kubernetes.io/projected/031cc8de-5bb9-4f1a-a40c-220fac42948d-kube-api-access-gjmj8\") pod \"payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.385000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.384970 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj7db\" (UniqueName: \"kubernetes.io/projected/00feb578-83bd-4297-bf32-2fc074d5a617-kube-api-access-cj7db\") pod \"payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.385811 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.385787 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00feb578-83bd-4297-bf32-2fc074d5a617-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.445803 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.445722 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj7db\" (UniqueName: \"kubernetes.io/projected/00feb578-83bd-4297-bf32-2fc074d5a617-kube-api-access-cj7db\") pod \"payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.484835 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.484805 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:15.486617 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.486587 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjmj8\" (UniqueName: \"kubernetes.io/projected/031cc8de-5bb9-4f1a-a40c-220fac42948d-kube-api-access-gjmj8\") pod \"payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.486756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.486713 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031cc8de-5bb9-4f1a-a40c-220fac42948d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.487382 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.487349 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031cc8de-5bb9-4f1a-a40c-220fac42948d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.543595 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.543568 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjmj8\" (UniqueName: \"kubernetes.io/projected/031cc8de-5bb9-4f1a-a40c-220fac42948d-kube-api-access-gjmj8\") pod \"payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.643610 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.643571 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:15.862919 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:15.862892 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9"] Oct 31 16:28:15.863274 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:28:15.863113 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00feb578_83bd_4297_bf32_2fc074d5a617.slice/crio-759f7f616043bffa2c941ee5962052601a5ec646648a22a9110f7d2c16664fb9 WatchSource:0}: Error finding container 759f7f616043bffa2c941ee5962052601a5ec646648a22a9110f7d2c16664fb9: Status 404 returned error can't find the container with id 759f7f616043bffa2c941ee5962052601a5ec646648a22a9110f7d2c16664fb9 Oct 31 16:28:16.019527 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:28:16.019496 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031cc8de_5bb9_4f1a_a40c_220fac42948d.slice/crio-44d357df8194d7eee58cf98b92f1de4e155daace051c6f7296da169281581134 WatchSource:0}: Error finding container 44d357df8194d7eee58cf98b92f1de4e155daace051c6f7296da169281581134: Status 404 returned error can't find the container with id 44d357df8194d7eee58cf98b92f1de4e155daace051c6f7296da169281581134 Oct 31 16:28:16.019527 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.019516 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj"] Oct 31 16:28:16.574630 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.574597 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" event={"ID":"00feb578-83bd-4297-bf32-2fc074d5a617","Type":"ContainerStarted","Data":"d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba"} Oct 31 16:28:16.574848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.574650 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" event={"ID":"00feb578-83bd-4297-bf32-2fc074d5a617","Type":"ContainerStarted","Data":"759f7f616043bffa2c941ee5962052601a5ec646648a22a9110f7d2c16664fb9"} Oct 31 16:28:16.574848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.574702 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:16.576334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.576311 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" event={"ID":"031cc8de-5bb9-4f1a-a40c-220fac42948d","Type":"ContainerStarted","Data":"8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba"} Oct 31 16:28:16.576485 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.576356 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" event={"ID":"031cc8de-5bb9-4f1a-a40c-220fac42948d","Type":"ContainerStarted","Data":"44d357df8194d7eee58cf98b92f1de4e155daace051c6f7296da169281581134"} Oct 31 16:28:16.576485 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.576393 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:16.688219 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.688156 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" podStartSLOduration=1.688143059 podStartE2EDuration="1.688143059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:16.687331409 +0000 UTC m=+6142.306882820" watchObservedRunningTime="2026-10-31 16:28:16.688143059 +0000 UTC m=+6142.307694471" Oct 31 16:28:16.806756 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:16.806680 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" podStartSLOduration=1.806665095 podStartE2EDuration="1.806665095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:16.799018479 +0000 UTC m=+6142.418569892" watchObservedRunningTime="2026-10-31 16:28:16.806665095 +0000 UTC m=+6142.426216506" Oct 31 16:28:17.490643 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:28:17.490610 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:28:17.942323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:17.942228 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:18.100315 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:18.100279 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:18.238097 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:18.238063 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:20.311607 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:20.311572 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:21.769703 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:21.769670 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:22.887524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:22.887496 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:23.214253 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:23.214210 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:25.821959 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:25.821921 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:28:27.049163 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:27.049132 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr"] Oct 31 16:28:27.052352 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:27.052330 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7p2fr"] Oct 31 16:28:27.584723 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:27.584693 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:28:27.584918 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:27.584767 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:28:27.831845 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:27.831806 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:28.505572 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:28.496234 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4164b92e-d477-42df-91d6-fa530701800d" path="/var/lib/kubelet/pods/4164b92e-d477-42df-91d6-fa530701800d/volumes" Oct 31 16:28:32.691401 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:32.691364 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:34.684678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:34.684642 4865 generic.go:356] "Generic (PLEG): container finished" podID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerID="ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f" exitCode=0 Oct 31 16:28:34.685106 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:34.684713 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" event={"ID":"fdbd65f1-4615-4104-a848-a23f7e681b79","Type":"ContainerDied","Data":"ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f"} Oct 31 16:28:34.685152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:34.685121 4865 scope.go:122] "RemoveContainer" containerID="ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f" Oct 31 16:28:34.686386 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:34.686361 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:28:34.686456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:34.686442 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:28:34.686531 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:34.686509 4865 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 16:28:35.692104 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:35.692017 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" event={"ID":"fdbd65f1-4615-4104-a848-a23f7e681b79","Type":"ContainerStarted","Data":"3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649"} Oct 31 16:28:36.420358 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:36.420317 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:36.516066 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:36.516026 4865 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" start-of-body= Oct 31 16:28:36.516359 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:36.516328 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" Oct 31 16:28:36.517393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:36.517370 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:36.967135 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:36.967101 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:37.119040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:37.118985 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:37.254107 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:37.254020 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:37.514549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:37.507805 4865 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" start-of-body= Oct 31 16:28:37.514549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:37.507865 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" Oct 31 16:28:37.924932 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:37.924848 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:38.524522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:38.515805 4865 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" start-of-body= Oct 31 16:28:38.524522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:38.515908 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" Oct 31 16:28:39.513496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.510867 4865 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" start-of-body= Oct 31 16:28:39.513496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.510945 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" Oct 31 16:28:39.534660 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.534629 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:39.853630 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.853554 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk"] Oct 31 16:28:39.856242 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.856222 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:39.922633 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.920133 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk"] Oct 31 16:28:39.939643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.939607 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:39.958564 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.958535 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bcf50e\"/\"kube-root-ca.crt\"" Oct 31 16:28:39.958738 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.958708 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bcf50e\"/\"default-dockercfg-cxxj5\"" Oct 31 16:28:39.958738 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.958712 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bcf50e\"/\"openshift-service-ca.crt\"" Oct 31 16:28:39.958880 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.958712 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bcf50e\"/\"simulator-single-main-bcf50e-kserve-self-signed-certs\"" Oct 31 16:28:39.961190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.961168 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-home\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:39.961322 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.961282 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-model-cache\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:39.961408 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.961391 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06935d65-3224-4443-8243-4a9d786138aa-tls-certs\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:39.961496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.961442 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-dshm\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:39.961579 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.961559 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trs4x\" (UniqueName: \"kubernetes.io/projected/06935d65-3224-4443-8243-4a9d786138aa-kube-api-access-trs4x\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:39.961627 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:39.961601 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-tmp-dir\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.062665 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.062633 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-model-cache\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.062893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.062696 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06935d65-3224-4443-8243-4a9d786138aa-tls-certs\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.062893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.062736 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-dshm\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.062893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.062773 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trs4x\" (UniqueName: \"kubernetes.io/projected/06935d65-3224-4443-8243-4a9d786138aa-kube-api-access-trs4x\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.062893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.062808 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-tmp-dir\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.062893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.062874 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-home\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.063208 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.063080 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-model-cache\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.063208 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.063165 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-home\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.063374 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.063347 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-tmp-dir\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.065360 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.065317 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-dshm\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.065555 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.065530 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06935d65-3224-4443-8243-4a9d786138aa-tls-certs\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.121192 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.121122 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trs4x\" (UniqueName: \"kubernetes.io/projected/06935d65-3224-4443-8243-4a9d786138aa-kube-api-access-trs4x\") pod \"simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.167227 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.167194 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:40.518644 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.518607 4865 patch_prober.go:29] interesting pod/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" start-of-body= Oct 31 16:28:40.519001 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.518943 4865 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.187:15021/healthz/ready\": dial tcp 10.132.1.187:15021: connect: connection refused" Oct 31 16:28:40.556678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.556652 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk"] Oct 31 16:28:40.557336 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:28:40.557310 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06935d65_3224_4443_8243_4a9d786138aa.slice/crio-85b1a96e85a7d38e5d74860f5d5f3372882934c612af749aa162028091756b44 WatchSource:0}: Error finding container 85b1a96e85a7d38e5d74860f5d5f3372882934c612af749aa162028091756b44: Status 404 returned error can't find the container with id 85b1a96e85a7d38e5d74860f5d5f3372882934c612af749aa162028091756b44 Oct 31 16:28:40.746459 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.746408 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" event={"ID":"06935d65-3224-4443-8243-4a9d786138aa","Type":"ContainerStarted","Data":"3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16"} Oct 31 16:28:40.746459 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.746487 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" event={"ID":"06935d65-3224-4443-8243-4a9d786138aa","Type":"ContainerStarted","Data":"85b1a96e85a7d38e5d74860f5d5f3372882934c612af749aa162028091756b44"} Oct 31 16:28:40.861673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:40.861553 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" podStartSLOduration=1.8615355820000001 podStartE2EDuration="1.861535582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:40.858294162 +0000 UTC m=+6166.477845574" watchObservedRunningTime="2026-10-31 16:28:40.861535582 +0000 UTC m=+6166.481086997" Oct 31 16:28:41.168103 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:41.168022 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:41.510430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:41.510401 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:41.511608 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:41.511581 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:28:42.913449 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:42.913412 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:44.429453 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:44.429418 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:28:47.489360 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:28:47.489324 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:28:47.698544 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:47.698424 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:50.045404 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:50.045370 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr"] Oct 31 16:28:50.094790 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:50.094758 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kpghr"] Oct 31 16:28:50.167425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:50.167395 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:50.180109 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:50.180072 4865 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:50.497098 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:50.497056 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536f940c-352e-4d05-9350-70cb16b4cf97" path="/var/lib/kubelet/pods/536f940c-352e-4d05-9350-70cb16b4cf97/volumes" Oct 31 16:28:50.811809 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:50.811723 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:28:52.496314 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:52.496253 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:57.306873 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:28:57.306839 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:01.322015 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:01.321983 4865 scope.go:122] "RemoveContainer" containerID="20e416c45946eaeecf27220705153092b1f414d1d31fc1ad4853535e05cd7012" Oct 31 16:29:02.132626 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:02.132594 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:03.049896 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:03.049864 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl"] Oct 31 16:29:03.099125 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:03.099095 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8r6dl"] Oct 31 16:29:04.494673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:04.494641 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c20188-d591-4ef0-9f6b-ce390d4b2de9" path="/var/lib/kubelet/pods/96c20188-d591-4ef0-9f6b-ce390d4b2de9/volumes" Oct 31 16:29:06.970139 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:06.970105 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:10.129068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:10.129031 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:29:11.704867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:11.704834 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:16.499749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:16.499718 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:17.489985 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:29:17.489947 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:29:20.646128 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:20.646091 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:21.220898 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:21.220857 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:22.133186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:22.133142 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:23.495481 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:23.495433 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:24.454736 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.454698 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn"] Oct 31 16:29:24.459176 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.459158 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.512484 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.512437 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn"] Oct 31 16:29:24.514206 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.514174 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.514350 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.514328 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfp9j\" (UniqueName: \"kubernetes.io/projected/a474017b-6bb5-4348-ac26-af537e6a6dfe-kube-api-access-jfp9j\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.514565 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.514536 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.557137 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.557093 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:29:24.557137 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.557135 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:29:24.557348 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.557152 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:29:24.560228 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.560211 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:29:24.615594 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.615569 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfp9j\" (UniqueName: \"kubernetes.io/projected/a474017b-6bb5-4348-ac26-af537e6a6dfe-kube-api-access-jfp9j\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.615749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.615644 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.615749 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.615676 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.616583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.616562 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.618709 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.618686 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.671681 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.671657 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfp9j\" (UniqueName: \"kubernetes.io/projected/a474017b-6bb5-4348-ac26-af537e6a6dfe-kube-api-access-jfp9j\") pod \"maas-api-revoke-sub-1e55dc89-jz4xn\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:24.772011 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:24.771983 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:25.144769 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:25.144740 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn"] Oct 31 16:29:25.145262 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:29:25.145240 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda474017b_6bb5_4348_ac26_af537e6a6dfe.slice/crio-d39b99dad9a18864c41af4fbd5ba22e2c6e91aca9e3f7352f5fc8f64c54cf2de WatchSource:0}: Error finding container d39b99dad9a18864c41af4fbd5ba22e2c6e91aca9e3f7352f5fc8f64c54cf2de: Status 404 returned error can't find the container with id d39b99dad9a18864c41af4fbd5ba22e2c6e91aca9e3f7352f5fc8f64c54cf2de Oct 31 16:29:25.990684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:25.990656 4865 generic.go:356] "Generic (PLEG): container finished" podID="a474017b-6bb5-4348-ac26-af537e6a6dfe" containerID="a7e9fca9b33d3fb657113e4b97f604327b7df35c6b053efbf56ae11c80dd580f" exitCode=0 Oct 31 16:29:25.991063 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:25.990731 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" event={"ID":"a474017b-6bb5-4348-ac26-af537e6a6dfe","Type":"ContainerDied","Data":"a7e9fca9b33d3fb657113e4b97f604327b7df35c6b053efbf56ae11c80dd580f"} Oct 31 16:29:25.991063 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:25.990775 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" event={"ID":"a474017b-6bb5-4348-ac26-af537e6a6dfe","Type":"ContainerStarted","Data":"d39b99dad9a18864c41af4fbd5ba22e2c6e91aca9e3f7352f5fc8f64c54cf2de"} Oct 31 16:29:26.012757 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:26.012731 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:27.138935 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.138910 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:27.238952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.238925 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-openshift-ca\") pod \"a474017b-6bb5-4348-ac26-af537e6a6dfe\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " Oct 31 16:29:27.239140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.238980 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a474017b-6bb5-4348-ac26-af537e6a6dfe-kube-api-access-jfp9j\" (UniqueName: \"kubernetes.io/projected/a474017b-6bb5-4348-ac26-af537e6a6dfe-kube-api-access-jfp9j\") pod \"a474017b-6bb5-4348-ac26-af537e6a6dfe\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " Oct 31 16:29:27.239140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.239122 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-xks-ca\") pod \"a474017b-6bb5-4348-ac26-af537e6a6dfe\" (UID: \"a474017b-6bb5-4348-ac26-af537e6a6dfe\") " Oct 31 16:29:27.239356 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.239335 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-openshift-ca" pod "a474017b-6bb5-4348-ac26-af537e6a6dfe" (UID: "a474017b-6bb5-4348-ac26-af537e6a6dfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:29:27.239500 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.239460 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.241241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.241208 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-xks-ca" pod "a474017b-6bb5-4348-ac26-af537e6a6dfe" (UID: "a474017b-6bb5-4348-ac26-af537e6a6dfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:29:27.241377 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.241275 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a474017b-6bb5-4348-ac26-af537e6a6dfe-kube-api-access-jfp9j" pod "a474017b-6bb5-4348-ac26-af537e6a6dfe" (UID: "a474017b-6bb5-4348-ac26-af537e6a6dfe"). InnerVolumeSpecName "kube-api-access-jfp9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:29:27.340653 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.340574 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfp9j\" (UniqueName: \"kubernetes.io/projected/a474017b-6bb5-4348-ac26-af537e6a6dfe-kube-api-access-jfp9j\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.340653 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:27.340603 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a474017b-6bb5-4348-ac26-af537e6a6dfe-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:28.003834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:28.003802 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" Oct 31 16:29:28.004029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:28.003837 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn" event={"ID":"a474017b-6bb5-4348-ac26-af537e6a6dfe","Type":"ContainerDied","Data":"d39b99dad9a18864c41af4fbd5ba22e2c6e91aca9e3f7352f5fc8f64c54cf2de"} Oct 31 16:29:28.004029 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:28.003887 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d39b99dad9a18864c41af4fbd5ba22e2c6e91aca9e3f7352f5fc8f64c54cf2de" Oct 31 16:29:30.803451 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:30.803412 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:35.606931 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:35.606894 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:36.149550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:36.149516 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:36.749937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:36.749907 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:38.045034 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:38.045000 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv"] Oct 31 16:29:38.094393 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:38.094353 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pntqv"] Oct 31 16:29:38.495537 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:38.495510 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0014dac4-f3f6-46fc-b31a-540d1c7807d7" path="/var/lib/kubelet/pods/0014dac4-f3f6-46fc-b31a-540d1c7807d7/volumes" Oct 31 16:29:39.404233 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:39.404199 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:29:40.412831 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:40.412795 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:41.126022 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:41.125989 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk"] Oct 31 16:29:41.219920 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:41.219889 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:42.367100 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:42.367065 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:43.159039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:43.159006 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj"] Oct 31 16:29:43.267208 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:43.267170 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9"] Oct 31 16:29:43.503969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:43.503938 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:43.620885 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:43.620847 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:44.932848 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:44.932810 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:47.489938 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:29:47.489906 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:29:47.751666 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:47.751619 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:48.159422 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:48.159333 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" podUID="031cc8de-5bb9-4f1a-a40c-220fac42948d" containerName="payload-processing" containerID="cri-o://8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba" gracePeriod=25 Oct 31 16:29:48.268008 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:48.267965 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" podUID="00feb578-83bd-4297-bf32-2fc074d5a617" containerName="payload-pre-processing" containerID="cri-o://d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba" gracePeriod=25 Oct 31 16:29:49.044798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:49.044761 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227"] Oct 31 16:29:49.094010 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:49.093974 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-6b227"] Oct 31 16:29:50.038186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:50.038137 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:29:50.508573 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:50.508538 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1159b497-11cf-4928-8102-5b3536f9776e" path="/var/lib/kubelet/pods/1159b497-11cf-4928-8102-5b3536f9776e/volumes" Oct 31 16:29:53.276281 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:53.276248 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:53.434976 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:53.434938 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:56.147460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.147424 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" podUID="06935d65-3224-4443-8243-4a9d786138aa" containerName="main" containerID="cri-o://3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16" gracePeriod=45 Oct 31 16:29:56.393598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.393572 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:29:56.523960 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.523921 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-tmp-dir\") pod \"06935d65-3224-4443-8243-4a9d786138aa\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " Oct 31 16:29:56.524144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524012 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-dshm\") pod \"06935d65-3224-4443-8243-4a9d786138aa\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " Oct 31 16:29:56.524144 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524058 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-model-cache\") pod \"06935d65-3224-4443-8243-4a9d786138aa\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " Oct 31 16:29:56.524285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524201 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-tmp-dir" pod "06935d65-3224-4443-8243-4a9d786138aa" (UID: "06935d65-3224-4443-8243-4a9d786138aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:56.524285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524239 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-home\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-home\") pod \"06935d65-3224-4443-8243-4a9d786138aa\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " Oct 31 16:29:56.524388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524301 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06935d65-3224-4443-8243-4a9d786138aa-kube-api-access-trs4x\" (UniqueName: \"kubernetes.io/projected/06935d65-3224-4443-8243-4a9d786138aa-kube-api-access-trs4x\") pod \"06935d65-3224-4443-8243-4a9d786138aa\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " Oct 31 16:29:56.524388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524347 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06935d65-3224-4443-8243-4a9d786138aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/06935d65-3224-4443-8243-4a9d786138aa-tls-certs\") pod \"06935d65-3224-4443-8243-4a9d786138aa\" (UID: \"06935d65-3224-4443-8243-4a9d786138aa\") " Oct 31 16:29:56.524388 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524362 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-model-cache" pod "06935d65-3224-4443-8243-4a9d786138aa" (UID: "06935d65-3224-4443-8243-4a9d786138aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:56.524583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524492 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-home" pod "06935d65-3224-4443-8243-4a9d786138aa" (UID: "06935d65-3224-4443-8243-4a9d786138aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:56.524776 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524722 4865 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-home\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:56.524776 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524772 4865 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-tmp-dir\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:56.524776 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.524790 4865 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-model-cache\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:56.526586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.526557 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-dshm" pod "06935d65-3224-4443-8243-4a9d786138aa" (UID: "06935d65-3224-4443-8243-4a9d786138aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:56.526908 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.526888 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06935d65-3224-4443-8243-4a9d786138aa-tls-certs" pod "06935d65-3224-4443-8243-4a9d786138aa" (UID: "06935d65-3224-4443-8243-4a9d786138aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:29:56.527103 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.527084 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06935d65-3224-4443-8243-4a9d786138aa-kube-api-access-trs4x" pod "06935d65-3224-4443-8243-4a9d786138aa" (UID: "06935d65-3224-4443-8243-4a9d786138aa"). InnerVolumeSpecName "kube-api-access-trs4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:29:56.625496 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.625439 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trs4x\" (UniqueName: \"kubernetes.io/projected/06935d65-3224-4443-8243-4a9d786138aa-kube-api-access-trs4x\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:56.625674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.625503 4865 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06935d65-3224-4443-8243-4a9d786138aa-tls-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:56.625674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:56.625518 4865 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06935d65-3224-4443-8243-4a9d786138aa-dshm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:29:57.164416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.164377 4865 generic.go:356] "Generic (PLEG): container finished" podID="06935d65-3224-4443-8243-4a9d786138aa" containerID="3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16" exitCode=0 Oct 31 16:29:57.164416 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.164415 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" Oct 31 16:29:57.164944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.164480 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" event={"ID":"06935d65-3224-4443-8243-4a9d786138aa","Type":"ContainerDied","Data":"3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16"} Oct 31 16:29:57.164944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.164518 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk" event={"ID":"06935d65-3224-4443-8243-4a9d786138aa","Type":"ContainerDied","Data":"85b1a96e85a7d38e5d74860f5d5f3372882934c612af749aa162028091756b44"} Oct 31 16:29:57.164944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.164537 4865 scope.go:122] "RemoveContainer" containerID="3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16" Oct 31 16:29:57.174424 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.174406 4865 scope.go:122] "RemoveContainer" containerID="3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16" Oct 31 16:29:57.174687 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:29:57.174669 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16\": container with ID starting with 3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16 not found: ID does not exist" containerID="3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16" Oct 31 16:29:57.174748 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.174695 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16"} err="failed to get container status \"3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16\": rpc error: code = NotFound desc = could not find container \"3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16\": container with ID starting with 3370126be980d998377d93be9d340a4d9133b40972695d0cb86c67ab03253b16 not found: ID does not exist" Oct 31 16:29:57.327431 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.327403 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk"] Oct 31 16:29:57.376549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.376522 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-bcf50e/simulator-single-main-bcf50e-kserve-64d45788c9-zr9pk"] Oct 31 16:29:57.583867 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.583831 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" podUID="00feb578-83bd-4297-bf32-2fc074d5a617" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:29:57.584069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:57.583836 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" podUID="031cc8de-5bb9-4f1a-a40c-220fac42948d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:29:58.494697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:29:58.494664 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06935d65-3224-4443-8243-4a9d786138aa" path="/var/lib/kubelet/pods/06935d65-3224-4443-8243-4a9d786138aa/volumes" Oct 31 16:30:00.155878 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.155846 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb"] Oct 31 16:30:00.156297 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.156283 4865 cpu_manager.go:414] "containerMap: removing container" podUID="a474017b-6bb5-4348-ac26-af537e6a6dfe" containerName="revoke-keys" Oct 31 16:30:00.156347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.156298 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a474017b-6bb5-4348-ac26-af537e6a6dfe" containerName="revoke-keys" Oct 31 16:30:00.156347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.156315 4865 cpu_manager.go:414] "containerMap: removing container" podUID="06935d65-3224-4443-8243-4a9d786138aa" containerName="main" Oct 31 16:30:00.156347 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.156320 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06935d65-3224-4443-8243-4a9d786138aa" containerName="main" Oct 31 16:30:00.156456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.156418 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="a474017b-6bb5-4348-ac26-af537e6a6dfe" containerName="revoke-keys" Oct 31 16:30:00.156456 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.156428 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="06935d65-3224-4443-8243-4a9d786138aa" containerName="main" Oct 31 16:30:00.160707 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.160688 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.214700 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.214673 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb"] Oct 31 16:30:00.259864 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.259836 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:30:00.260007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.259876 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:30:00.260007 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.259965 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:30:00.260319 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.260303 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a8d63-a630-469d-9cc8-85d850bcddcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891070-bm7lb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.260358 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.260326 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc5zp\" (UniqueName: \"kubernetes.io/projected/ec1a8d63-a630-469d-9cc8-85d850bcddcb-kube-api-access-pc5zp\") pod \"maas-api-key-cleanup-29891070-bm7lb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.361151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.361120 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a8d63-a630-469d-9cc8-85d850bcddcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891070-bm7lb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.361151 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.361151 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5zp\" (UniqueName: \"kubernetes.io/projected/ec1a8d63-a630-469d-9cc8-85d850bcddcb-kube-api-access-pc5zp\") pod \"maas-api-key-cleanup-29891070-bm7lb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.361762 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.361744 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a8d63-a630-469d-9cc8-85d850bcddcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891070-bm7lb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.417729 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.417666 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5zp\" (UniqueName: \"kubernetes.io/projected/ec1a8d63-a630-469d-9cc8-85d850bcddcb-kube-api-access-pc5zp\") pod \"maas-api-key-cleanup-29891070-bm7lb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.473017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.472991 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:00.843970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:00.843943 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb"] Oct 31 16:30:00.844760 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:30:00.844735 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1a8d63_a630_469d_9cc8_85d850bcddcb.slice/crio-079311b205e3c956bf82c97cf7333183a6f6aca17c9ab8ffca3b71708ef774f7 WatchSource:0}: Error finding container 079311b205e3c956bf82c97cf7333183a6f6aca17c9ab8ffca3b71708ef774f7: Status 404 returned error can't find the container with id 079311b205e3c956bf82c97cf7333183a6f6aca17c9ab8ffca3b71708ef774f7 Oct 31 16:30:01.188604 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:01.188575 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" event={"ID":"ec1a8d63-a630-469d-9cc8-85d850bcddcb","Type":"ContainerStarted","Data":"27b22d98d1c877c19a74b7ce4c42a3af7c69dd76c90b2f351b64883df74021b8"} Oct 31 16:30:01.188929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:01.188623 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" event={"ID":"ec1a8d63-a630-469d-9cc8-85d850bcddcb","Type":"ContainerStarted","Data":"079311b205e3c956bf82c97cf7333183a6f6aca17c9ab8ffca3b71708ef774f7"} Oct 31 16:30:01.300767 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:01.300712 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" podStartSLOduration=1.30069798 podStartE2EDuration="1.30069798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:01.299932493 +0000 UTC m=+6246.919483904" watchObservedRunningTime="2026-10-31 16:30:01.30069798 +0000 UTC m=+6246.920249403" Oct 31 16:30:01.348430 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:01.348400 4865 scope.go:122] "RemoveContainer" containerID="bcd2fdc6798c15dbc60777079f8e023bfcca1eda208f7b0c03694601447ae3d2" Oct 31 16:30:01.357434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:01.357416 4865 scope.go:122] "RemoveContainer" containerID="dab3155270a1ad1be0ea4b3fbff92b981ecda871a7ca8ee2eecbc5b16108409f" Oct 31 16:30:01.365649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:01.365624 4865 scope.go:122] "RemoveContainer" containerID="e90072ce55480b7970ae289b5b767135fb9813d5ca2bcd893a20cf0a93b0aa14" Oct 31 16:30:02.195699 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:02.195660 4865 generic.go:356] "Generic (PLEG): container finished" podID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerID="27b22d98d1c877c19a74b7ce4c42a3af7c69dd76c90b2f351b64883df74021b8" exitCode=60 Oct 31 16:30:02.196126 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:02.195749 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" event={"ID":"ec1a8d63-a630-469d-9cc8-85d850bcddcb","Type":"ContainerDied","Data":"27b22d98d1c877c19a74b7ce4c42a3af7c69dd76c90b2f351b64883df74021b8"} Oct 31 16:30:02.196126 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:02.195945 4865 scope.go:122] "RemoveContainer" containerID="27b22d98d1c877c19a74b7ce4c42a3af7c69dd76c90b2f351b64883df74021b8" Oct 31 16:30:03.202978 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:03.202899 4865 generic.go:356] "Generic (PLEG): container finished" podID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerID="43117ee5ee7bbee68eefa4b720480754509aacc1d9ee6de714c9a23b01ea954f" exitCode=60 Oct 31 16:30:03.202978 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:03.202966 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" event={"ID":"ec1a8d63-a630-469d-9cc8-85d850bcddcb","Type":"ContainerDied","Data":"43117ee5ee7bbee68eefa4b720480754509aacc1d9ee6de714c9a23b01ea954f"} Oct 31 16:30:03.203517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:03.203011 4865 scope.go:122] "RemoveContainer" containerID="27b22d98d1c877c19a74b7ce4c42a3af7c69dd76c90b2f351b64883df74021b8" Oct 31 16:30:03.203517 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:03.203176 4865 scope.go:122] "RemoveContainer" containerID="43117ee5ee7bbee68eefa4b720480754509aacc1d9ee6de714c9a23b01ea954f" Oct 31 16:30:03.203517 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:03.203430 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891070-bm7lb_odh-ai-gateway-infra(ec1a8d63-a630-469d-9cc8-85d850bcddcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" Oct 31 16:30:04.210414 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:04.210388 4865 scope.go:122] "RemoveContainer" containerID="43117ee5ee7bbee68eefa4b720480754509aacc1d9ee6de714c9a23b01ea954f" Oct 31 16:30:04.210857 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:04.210616 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891070-bm7lb_odh-ai-gateway-infra(ec1a8d63-a630-469d-9cc8-85d850bcddcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" Oct 31 16:30:07.583241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:07.583154 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" podUID="031cc8de-5bb9-4f1a-a40c-220fac42948d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:30:07.583241 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:07.583191 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" podUID="00feb578-83bd-4297-bf32-2fc074d5a617" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:30:08.403706 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.403678 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:30:08.437287 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.437213 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031cc8de-5bb9-4f1a-a40c-220fac42948d-kube-api-access-gjmj8\" (UniqueName: \"kubernetes.io/projected/031cc8de-5bb9-4f1a-a40c-220fac42948d-kube-api-access-gjmj8\") pod \"031cc8de-5bb9-4f1a-a40c-220fac42948d\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " Oct 31 16:30:08.437450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.437331 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031cc8de-5bb9-4f1a-a40c-220fac42948d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031cc8de-5bb9-4f1a-a40c-220fac42948d-plugins-config-volume\") pod \"031cc8de-5bb9-4f1a-a40c-220fac42948d\" (UID: \"031cc8de-5bb9-4f1a-a40c-220fac42948d\") " Oct 31 16:30:08.438101 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.437693 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031cc8de-5bb9-4f1a-a40c-220fac42948d-plugins-config-volume" pod "031cc8de-5bb9-4f1a-a40c-220fac42948d" (UID: "031cc8de-5bb9-4f1a-a40c-220fac42948d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:30:08.439988 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.439956 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031cc8de-5bb9-4f1a-a40c-220fac42948d-kube-api-access-gjmj8" pod "031cc8de-5bb9-4f1a-a40c-220fac42948d" (UID: "031cc8de-5bb9-4f1a-a40c-220fac42948d"). InnerVolumeSpecName "kube-api-access-gjmj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:30:08.517349 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.517317 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:30:08.538024 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.537990 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00feb578-83bd-4297-bf32-2fc074d5a617-kube-api-access-cj7db\" (UniqueName: \"kubernetes.io/projected/00feb578-83bd-4297-bf32-2fc074d5a617-kube-api-access-cj7db\") pod \"00feb578-83bd-4297-bf32-2fc074d5a617\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " Oct 31 16:30:08.538189 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.538169 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00feb578-83bd-4297-bf32-2fc074d5a617-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00feb578-83bd-4297-bf32-2fc074d5a617-plugins-config-volume\") pod \"00feb578-83bd-4297-bf32-2fc074d5a617\" (UID: \"00feb578-83bd-4297-bf32-2fc074d5a617\") " Oct 31 16:30:08.538487 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.538456 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031cc8de-5bb9-4f1a-a40c-220fac42948d-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:08.538554 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.538492 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjmj8\" (UniqueName: \"kubernetes.io/projected/031cc8de-5bb9-4f1a-a40c-220fac42948d-kube-api-access-gjmj8\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:08.538591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.538548 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00feb578-83bd-4297-bf32-2fc074d5a617-plugins-config-volume" pod "00feb578-83bd-4297-bf32-2fc074d5a617" (UID: "00feb578-83bd-4297-bf32-2fc074d5a617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:30:08.540351 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.540319 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00feb578-83bd-4297-bf32-2fc074d5a617-kube-api-access-cj7db" pod "00feb578-83bd-4297-bf32-2fc074d5a617" (UID: "00feb578-83bd-4297-bf32-2fc074d5a617"). InnerVolumeSpecName "kube-api-access-cj7db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:30:08.639441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.639411 4865 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00feb578-83bd-4297-bf32-2fc074d5a617-plugins-config-volume\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:08.639441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:08.639437 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj7db\" (UniqueName: \"kubernetes.io/projected/00feb578-83bd-4297-bf32-2fc074d5a617-kube-api-access-cj7db\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:09.237041 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.237008 4865 generic.go:356] "Generic (PLEG): container finished" podID="00feb578-83bd-4297-bf32-2fc074d5a617" containerID="d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba" exitCode=0 Oct 31 16:30:09.237217 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.237057 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" Oct 31 16:30:09.237217 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.237093 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" event={"ID":"00feb578-83bd-4297-bf32-2fc074d5a617","Type":"ContainerDied","Data":"d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba"} Oct 31 16:30:09.237217 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.237143 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9" event={"ID":"00feb578-83bd-4297-bf32-2fc074d5a617","Type":"ContainerDied","Data":"759f7f616043bffa2c941ee5962052601a5ec646648a22a9110f7d2c16664fb9"} Oct 31 16:30:09.237217 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.237170 4865 scope.go:122] "RemoveContainer" containerID="d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba" Oct 31 16:30:09.238678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.238551 4865 generic.go:356] "Generic (PLEG): container finished" podID="031cc8de-5bb9-4f1a-a40c-220fac42948d" containerID="8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba" exitCode=0 Oct 31 16:30:09.238678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.238601 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" Oct 31 16:30:09.238678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.238597 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" event={"ID":"031cc8de-5bb9-4f1a-a40c-220fac42948d","Type":"ContainerDied","Data":"8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba"} Oct 31 16:30:09.238678 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.238648 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj" event={"ID":"031cc8de-5bb9-4f1a-a40c-220fac42948d","Type":"ContainerDied","Data":"44d357df8194d7eee58cf98b92f1de4e155daace051c6f7296da169281581134"} Oct 31 16:30:09.247716 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.247697 4865 scope.go:122] "RemoveContainer" containerID="d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba" Oct 31 16:30:09.247971 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:09.247954 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba\": container with ID starting with d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba not found: ID does not exist" containerID="d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba" Oct 31 16:30:09.248044 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.247977 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba"} err="failed to get container status \"d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba\": rpc error: code = NotFound desc = could not find container \"d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba\": container with ID starting with d54c0eeb6168fddafb616384bb704859621db32509d2f617474d57343ebbdcba not found: ID does not exist" Oct 31 16:30:09.248044 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.247994 4865 scope.go:122] "RemoveContainer" containerID="8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba" Oct 31 16:30:09.255871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.255855 4865 scope.go:122] "RemoveContainer" containerID="8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba" Oct 31 16:30:09.256126 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:09.256109 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba\": container with ID starting with 8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba not found: ID does not exist" containerID="8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba" Oct 31 16:30:09.256171 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.256133 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba"} err="failed to get container status \"8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba\": rpc error: code = NotFound desc = could not find container \"8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba\": container with ID starting with 8fa4d028b530aabcd64e42cc9cd67e5f80b0d6b6785aba90f5e49a58dd4c8aba not found: ID does not exist" Oct 31 16:30:09.409643 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.409610 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj"] Oct 31 16:30:09.410630 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.410607 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcf50e-784486849zkbkj"] Oct 31 16:30:09.511842 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.511812 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9"] Oct 31 16:30:09.514926 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:09.514906 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bcf50e-699dfp89d9"] Oct 31 16:30:10.495581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:10.495550 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00feb578-83bd-4297-bf32-2fc074d5a617" path="/var/lib/kubelet/pods/00feb578-83bd-4297-bf32-2fc074d5a617/volumes" Oct 31 16:30:10.495995 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:10.495945 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031cc8de-5bb9-4f1a-a40c-220fac42948d" path="/var/lib/kubelet/pods/031cc8de-5bb9-4f1a-a40c-220fac42948d/volumes" Oct 31 16:30:15.200645 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:15.200609 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:15.489367 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:15.489342 4865 scope.go:122] "RemoveContainer" containerID="43117ee5ee7bbee68eefa4b720480754509aacc1d9ee6de714c9a23b01ea954f" Oct 31 16:30:16.279068 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:16.279030 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" event={"ID":"ec1a8d63-a630-469d-9cc8-85d850bcddcb","Type":"ContainerStarted","Data":"a309e6509349fd586c500da1142c17323ce7d76aa48a558c49c55dc76996d228"} Oct 31 16:30:17.286837 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:17.286802 4865 generic.go:356] "Generic (PLEG): container finished" podID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerID="a309e6509349fd586c500da1142c17323ce7d76aa48a558c49c55dc76996d228" exitCode=60 Oct 31 16:30:17.287285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:17.286874 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" event={"ID":"ec1a8d63-a630-469d-9cc8-85d850bcddcb","Type":"ContainerDied","Data":"a309e6509349fd586c500da1142c17323ce7d76aa48a558c49c55dc76996d228"} Oct 31 16:30:17.287285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:17.286936 4865 scope.go:122] "RemoveContainer" containerID="43117ee5ee7bbee68eefa4b720480754509aacc1d9ee6de714c9a23b01ea954f" Oct 31 16:30:17.287285 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:17.287260 4865 scope.go:122] "RemoveContainer" containerID="a309e6509349fd586c500da1142c17323ce7d76aa48a558c49c55dc76996d228" Oct 31 16:30:17.287536 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:17.287516 4865 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891070-bm7lb_odh-ai-gateway-infra(ec1a8d63-a630-469d-9cc8-85d850bcddcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" Oct 31 16:30:17.408127 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:17.408099 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb"] Oct 31 16:30:17.491152 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:17.491113 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:30:17.501937 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:17.501904 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:18.181869 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.181831 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:18.432186 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.432127 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:18.535226 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.535193 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1a8d63-a630-469d-9cc8-85d850bcddcb-kube-api-access-pc5zp\" (UniqueName: \"kubernetes.io/projected/ec1a8d63-a630-469d-9cc8-85d850bcddcb-kube-api-access-pc5zp\") pod \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " Oct 31 16:30:18.535413 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.535243 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1a8d63-a630-469d-9cc8-85d850bcddcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a8d63-a630-469d-9cc8-85d850bcddcb-maas-api-service-ca\") pod \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\" (UID: \"ec1a8d63-a630-469d-9cc8-85d850bcddcb\") " Oct 31 16:30:18.535710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.535684 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1a8d63-a630-469d-9cc8-85d850bcddcb-maas-api-service-ca" pod "ec1a8d63-a630-469d-9cc8-85d850bcddcb" (UID: "ec1a8d63-a630-469d-9cc8-85d850bcddcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:30:18.537528 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.537504 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1a8d63-a630-469d-9cc8-85d850bcddcb-kube-api-access-pc5zp" pod "ec1a8d63-a630-469d-9cc8-85d850bcddcb" (UID: "ec1a8d63-a630-469d-9cc8-85d850bcddcb"). InnerVolumeSpecName "kube-api-access-pc5zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:30:18.636483 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.636438 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc5zp\" (UniqueName: \"kubernetes.io/projected/ec1a8d63-a630-469d-9cc8-85d850bcddcb-kube-api-access-pc5zp\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:18.636483 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:18.636482 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a8d63-a630-469d-9cc8-85d850bcddcb-maas-api-service-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:19.300759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:19.300720 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" event={"ID":"ec1a8d63-a630-469d-9cc8-85d850bcddcb","Type":"ContainerDied","Data":"079311b205e3c956bf82c97cf7333183a6f6aca17c9ab8ffca3b71708ef774f7"} Oct 31 16:30:19.300759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:19.300738 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb" Oct 31 16:30:19.301053 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:19.300785 4865 scope.go:122] "RemoveContainer" containerID="a309e6509349fd586c500da1142c17323ce7d76aa48a558c49c55dc76996d228" Oct 31 16:30:19.463697 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:19.463659 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb"] Oct 31 16:30:19.513050 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:19.513019 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-bm7lb"] Oct 31 16:30:20.000203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:20.000171 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:20.495379 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:20.495343 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" path="/var/lib/kubelet/pods/ec1a8d63-a630-469d-9cc8-85d850bcddcb/volumes" Oct 31 16:30:20.608244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:20.608208 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:20.614803 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:20.614780 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs"] Oct 31 16:30:20.615021 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:20.614989 4865 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" containerID="cri-o://3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649" gracePeriod=30 Oct 31 16:30:20.992827 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:20.992793 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:22.445887 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:22.445853 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:23.566799 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:23.566757 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:23.688818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:23.688784 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:23.873902 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:23.873815 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:24.044076 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:24.044049 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4"] Oct 31 16:30:24.093989 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:24.093951 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wlfm4"] Oct 31 16:30:24.495195 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:24.495161 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1" path="/var/lib/kubelet/pods/e8630f5c-f75e-4a8d-b8ef-4d1edf83a8c1/volumes" Oct 31 16:30:25.874522 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:25.874495 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:30:26.013606 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013577 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-token\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.013785 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013637 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-socket\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.013785 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013748 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbd65f1-4615-4104-a848-a23f7e681b79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd65f1-4615-4104-a848-a23f7e681b79-istiod-ca-cert\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.013900 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013790 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-certs\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.013900 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013853 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-data\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.014033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013896 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-credential-socket\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.014033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013928 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-podinfo\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.014033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013935 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-socket" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:30:26.014033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.013963 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-envoy\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.014033 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014009 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-kube-api-access-252hh\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-kube-api-access-252hh\") pod \"fdbd65f1-4615-4104-a848-a23f7e681b79\" (UID: \"fdbd65f1-4615-4104-a848-a23f7e681b79\") " Oct 31 16:30:26.014350 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014128 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-certs" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:30:26.014350 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014135 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbd65f1-4615-4104-a848-a23f7e681b79-istiod-ca-cert" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:30:26.014350 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014141 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-data" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:30:26.014350 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014153 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-credential-socket" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:30:26.014570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014420 4865 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.014570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014451 4865 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd65f1-4615-4104-a848-a23f7e681b79-istiod-ca-cert\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.014570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014460 4865 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-workload-certs\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.014570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014500 4865 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-data\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.014570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.014510 4865 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-credential-socket\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.016052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.016032 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-token" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:30:26.016588 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.016570 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-kube-api-access-252hh" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "kube-api-access-252hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:30:26.016739 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.016717 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-podinfo" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:30:26.016871 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.016735 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-envoy" pod "fdbd65f1-4615-4104-a848-a23f7e681b79" (UID: "fdbd65f1-4615-4104-a848-a23f7e681b79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:30:26.115947 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.115904 4865 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-podinfo\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.115947 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.115949 4865 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-envoy\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.116203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.115988 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-252hh\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-kube-api-access-252hh\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.116203 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.116006 4865 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd65f1-4615-4104-a848-a23f7e681b79-istio-token\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:26.348412 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.348323 4865 generic.go:356] "Generic (PLEG): container finished" podID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerID="3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649" exitCode=0 Oct 31 16:30:26.348412 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.348391 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" Oct 31 16:30:26.348412 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.348395 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" event={"ID":"fdbd65f1-4615-4104-a848-a23f7e681b79","Type":"ContainerDied","Data":"3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649"} Oct 31 16:30:26.348804 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.348497 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs" event={"ID":"fdbd65f1-4615-4104-a848-a23f7e681b79","Type":"ContainerDied","Data":"41b4d2cc2a46bcf7948293dab3b944a21bbad21d2fd582c3e52cb95911d3f142"} Oct 31 16:30:26.348804 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.348522 4865 scope.go:122] "RemoveContainer" containerID="3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649" Oct 31 16:30:26.358183 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.358164 4865 scope.go:122] "RemoveContainer" containerID="ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f" Oct 31 16:30:26.366846 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.366828 4865 scope.go:122] "RemoveContainer" containerID="3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649" Oct 31 16:30:26.367125 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:26.367107 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649\": container with ID starting with 3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649 not found: ID does not exist" containerID="3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649" Oct 31 16:30:26.367205 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.367138 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649"} err="failed to get container status \"3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649\": rpc error: code = NotFound desc = could not find container \"3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649\": container with ID starting with 3ea5396bfb6a4f140536472e3de18299fd0564ddd14d527a32c65236be1d8649 not found: ID does not exist" Oct 31 16:30:26.367205 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.367161 4865 scope.go:122] "RemoveContainer" containerID="ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f" Oct 31 16:30:26.367408 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:26.367390 4865 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f\": container with ID starting with ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f not found: ID does not exist" containerID="ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f" Oct 31 16:30:26.367503 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.367418 4865 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f"} err="failed to get container status \"ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f\": rpc error: code = NotFound desc = could not find container \"ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f\": container with ID starting with ca397ce0dda82fa9b4a2e5d58e98bc7b56ff89f97372d25726015abc745d2f5f not found: ID does not exist" Oct 31 16:30:26.512331 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.512298 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs"] Oct 31 16:30:26.515244 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:26.515222 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bcf50e-openshift-default-78bc96596fdmhcs"] Oct 31 16:30:28.496038 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:28.496006 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" path="/var/lib/kubelet/pods/fdbd65f1-4615-4104-a848-a23f7e681b79/volumes" Oct 31 16:30:29.576048 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.576013 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj"] Oct 31 16:30:29.579423 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.579398 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mh8pj"] Oct 31 16:30:29.694992 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.694957 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth"] Oct 31 16:30:29.695361 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695348 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" Oct 31 16:30:29.695409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695362 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" Oct 31 16:30:29.695409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695373 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:30:29.695409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695379 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:30:29.695409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695391 4865 cpu_manager.go:414] "containerMap: removing container" podUID="00feb578-83bd-4297-bf32-2fc074d5a617" containerName="payload-pre-processing" Oct 31 16:30:29.695409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695398 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00feb578-83bd-4297-bf32-2fc074d5a617" containerName="payload-pre-processing" Oct 31 16:30:29.695409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695405 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:30:29.695409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695410 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:30:29.695673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695431 4865 cpu_manager.go:414] "containerMap: removing container" podUID="031cc8de-5bb9-4f1a-a40c-220fac42948d" containerName="payload-processing" Oct 31 16:30:29.695673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695436 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031cc8de-5bb9-4f1a-a40c-220fac42948d" containerName="payload-processing" Oct 31 16:30:29.695673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695522 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="031cc8de-5bb9-4f1a-a40c-220fac42948d" containerName="payload-processing" Oct 31 16:30:29.695673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695531 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" Oct 31 16:30:29.695673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695540 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:30:29.695673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695546 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="00feb578-83bd-4297-bf32-2fc074d5a617" containerName="payload-pre-processing" Oct 31 16:30:29.695673 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.695552 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:30:29.700158 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.700139 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.752547 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.752517 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:30:29.753070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.752536 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:30:29.753070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.752643 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:30:29.753070 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.752721 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:30:29.754849 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.754827 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth"] Oct 31 16:30:29.855663 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.855550 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfknc\" (UniqueName: \"kubernetes.io/projected/8d37b74f-2921-41c5-9468-5b2e6824cbef-kube-api-access-tfknc\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.855850 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.855699 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.855850 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.855779 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.956396 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.956355 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.956624 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.956425 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfknc\" (UniqueName: \"kubernetes.io/projected/8d37b74f-2921-41c5-9468-5b2e6824cbef-kube-api-access-tfknc\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.956624 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.956535 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.957031 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.957009 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:29.959129 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:29.959102 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:30.013558 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:30.013527 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfknc\" (UniqueName: \"kubernetes.io/projected/8d37b74f-2921-41c5-9468-5b2e6824cbef-kube-api-access-tfknc\") pod \"maas-api-revoke-sub-3d4156f1-2hsth\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:30.312963 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:30.312930 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:30.499970 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:30.499930 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fb1b70-1d98-41a8-8896-73b931915953" path="/var/lib/kubelet/pods/79fb1b70-1d98-41a8-8896-73b931915953/volumes" Oct 31 16:30:30.684190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:30.684161 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth"] Oct 31 16:30:30.685417 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:30:30.685390 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d37b74f_2921_41c5_9468_5b2e6824cbef.slice/crio-076906e20ee30420ac543eea63d2082ba0b5fda0b3e36336fb08b8d9f51b0496 WatchSource:0}: Error finding container 076906e20ee30420ac543eea63d2082ba0b5fda0b3e36336fb08b8d9f51b0496: Status 404 returned error can't find the container with id 076906e20ee30420ac543eea63d2082ba0b5fda0b3e36336fb08b8d9f51b0496 Oct 31 16:30:31.377236 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:31.377145 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" event={"ID":"8d37b74f-2921-41c5-9468-5b2e6824cbef","Type":"ContainerStarted","Data":"df5f2feed189a6f478a741026a71ccacdff9e3467e60c53152e4e4c1c94e9ffa"} Oct 31 16:30:31.377236 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:31.377211 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" event={"ID":"8d37b74f-2921-41c5-9468-5b2e6824cbef","Type":"ContainerStarted","Data":"076906e20ee30420ac543eea63d2082ba0b5fda0b3e36336fb08b8d9f51b0496"} Oct 31 16:30:31.490490 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:31.490411 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" podStartSLOduration=2.490397657 podStartE2EDuration="2.490397657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:31.489963983 +0000 UTC m=+6277.109515395" watchObservedRunningTime="2026-10-31 16:30:31.490397657 +0000 UTC m=+6277.109949068" Oct 31 16:30:32.384747 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:32.384707 4865 generic.go:356] "Generic (PLEG): container finished" podID="8d37b74f-2921-41c5-9468-5b2e6824cbef" containerID="df5f2feed189a6f478a741026a71ccacdff9e3467e60c53152e4e4c1c94e9ffa" exitCode=0 Oct 31 16:30:32.385183 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:32.384788 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" event={"ID":"8d37b74f-2921-41c5-9468-5b2e6824cbef","Type":"ContainerDied","Data":"df5f2feed189a6f478a741026a71ccacdff9e3467e60c53152e4e4c1c94e9ffa"} Oct 31 16:30:33.532381 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.532355 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:33.696167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.696081 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-xks-ca\") pod \"8d37b74f-2921-41c5-9468-5b2e6824cbef\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " Oct 31 16:30:33.696167 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.696135 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d37b74f-2921-41c5-9468-5b2e6824cbef-kube-api-access-tfknc\" (UniqueName: \"kubernetes.io/projected/8d37b74f-2921-41c5-9468-5b2e6824cbef-kube-api-access-tfknc\") pod \"8d37b74f-2921-41c5-9468-5b2e6824cbef\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " Oct 31 16:30:33.696383 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.696230 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-openshift-ca\") pod \"8d37b74f-2921-41c5-9468-5b2e6824cbef\" (UID: \"8d37b74f-2921-41c5-9468-5b2e6824cbef\") " Oct 31 16:30:33.696609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.696584 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-openshift-ca" pod "8d37b74f-2921-41c5-9468-5b2e6824cbef" (UID: "8d37b74f-2921-41c5-9468-5b2e6824cbef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:30:33.698427 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.698397 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-xks-ca" pod "8d37b74f-2921-41c5-9468-5b2e6824cbef" (UID: "8d37b74f-2921-41c5-9468-5b2e6824cbef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:30:33.698552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.698523 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d37b74f-2921-41c5-9468-5b2e6824cbef-kube-api-access-tfknc" pod "8d37b74f-2921-41c5-9468-5b2e6824cbef" (UID: "8d37b74f-2921-41c5-9468-5b2e6824cbef"). InnerVolumeSpecName "kube-api-access-tfknc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:30:33.797862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.797824 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:33.797862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.797856 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfknc\" (UniqueName: \"kubernetes.io/projected/8d37b74f-2921-41c5-9468-5b2e6824cbef-kube-api-access-tfknc\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:33.797862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:33.797867 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d37b74f-2921-41c5-9468-5b2e6824cbef-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:30:34.396812 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:34.396771 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" event={"ID":"8d37b74f-2921-41c5-9468-5b2e6824cbef","Type":"ContainerDied","Data":"076906e20ee30420ac543eea63d2082ba0b5fda0b3e36336fb08b8d9f51b0496"} Oct 31 16:30:34.397000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:34.396829 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="076906e20ee30420ac543eea63d2082ba0b5fda0b3e36336fb08b8d9f51b0496" Oct 31 16:30:34.397000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:34.396787 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth" Oct 31 16:30:43.044921 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:43.044880 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw"] Oct 31 16:30:43.093929 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:43.093896 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cpxqw"] Oct 31 16:30:44.495640 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:44.495606 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2233fabd-8b91-4137-86e7-08d5ac7ca568" path="/var/lib/kubelet/pods/2233fabd-8b91-4137-86e7-08d5ac7ca568/volumes" Oct 31 16:30:47.489367 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:30:47.489330 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:30:56.511591 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:56.511564 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:30:56.520104 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:56.520079 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:30:58.898973 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:58.898943 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:30:58.907557 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:30:58.907532 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:31:01.400581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:01.400546 4865 scope.go:122] "RemoveContainer" containerID="0e7b5e583050305ce564d8d54604292789331a0f7689aebc3aa31a4303d31ae6" Oct 31 16:31:01.409706 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:01.409682 4865 scope.go:122] "RemoveContainer" containerID="750a56c7fcd07810af6b5866eaa4db01c4baffe5b968b09c0fd6a9e6a5c7b683" Oct 31 16:31:04.046814 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:04.046774 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl"] Oct 31 16:31:04.101217 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:04.101179 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-4n4kl"] Oct 31 16:31:04.495351 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:04.495316 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fa45d8-a237-44d6-869d-4b4e35990299" path="/var/lib/kubelet/pods/a4fa45d8-a237-44d6-869d-4b4e35990299/volumes" Oct 31 16:31:13.743148 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:13.743110 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:15.230927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:15.230890 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:17.489902 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:31:17.489863 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:31:21.345798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:21.345765 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:22.047326 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:22.047293 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm"] Oct 31 16:31:22.096407 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:22.096375 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mqtwm"] Oct 31 16:31:22.494625 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:22.494595 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8beee6d1-08f8-48b8-9a1d-4ab94ea0a265" path="/var/lib/kubelet/pods/8beee6d1-08f8-48b8-9a1d-4ab94ea0a265/volumes" Oct 31 16:31:22.793449 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:22.793361 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:24.187563 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:31:24.187520 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:47.489611 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:31:47.489570 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:32:01.451516 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:01.451458 4865 scope.go:122] "RemoveContainer" containerID="00f480bc0db9cbe62b1c216114b9e5e663a53600b9fa29518bef71facb622aa4" Oct 31 16:32:01.460435 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:01.460415 4865 scope.go:122] "RemoveContainer" containerID="6bbfc132f72d876d09c4b021e387998522bce0d6ad24a069ce14e56a100acb5e" Oct 31 16:32:01.468987 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:01.468962 4865 scope.go:122] "RemoveContainer" containerID="34bd4387ad5bc2bc424d85eb280625f08900bb30b7b22d2d1165101d1abf407f" Oct 31 16:32:14.533888 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.533826 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:14.588304 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588271 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv"] Oct 31 16:32:14.588758 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588741 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:32:14.588865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588760 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:32:14.588865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588790 4865 cpu_manager.go:414] "containerMap: removing container" podUID="8d37b74f-2921-41c5-9468-5b2e6824cbef" containerName="revoke-keys" Oct 31 16:32:14.588865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588796 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d37b74f-2921-41c5-9468-5b2e6824cbef" containerName="revoke-keys" Oct 31 16:32:14.588865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588823 4865 cpu_manager.go:414] "containerMap: removing container" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" Oct 31 16:32:14.588865 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588829 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" Oct 31 16:32:14.589190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588894 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d37b74f-2921-41c5-9468-5b2e6824cbef" containerName="revoke-keys" Oct 31 16:32:14.589190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588906 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbd65f1-4615-4104-a848-a23f7e681b79" containerName="istio-proxy" Oct 31 16:32:14.589190 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.588917 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1a8d63-a630-469d-9cc8-85d850bcddcb" containerName="cleanup" Oct 31 16:32:14.593858 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.593842 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.647677 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.647649 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv"] Oct 31 16:32:14.658034 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.658010 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e367f06-ed74-4810-9703-a762eeaae827-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.658152 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.658080 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjdn\" (UniqueName: \"kubernetes.io/projected/2e367f06-ed74-4810-9703-a762eeaae827-kube-api-access-bvjdn\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.658201 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.658160 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e367f06-ed74-4810-9703-a762eeaae827-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.659713 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.659687 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 16:32:14.672938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.672911 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:14.692592 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.692562 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 16:32:14.758744 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.758716 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjdn\" (UniqueName: \"kubernetes.io/projected/2e367f06-ed74-4810-9703-a762eeaae827-kube-api-access-bvjdn\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.758905 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.758756 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e367f06-ed74-4810-9703-a762eeaae827-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.758905 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.758812 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e367f06-ed74-4810-9703-a762eeaae827-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.759636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.759572 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e367f06-ed74-4810-9703-a762eeaae827-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.761487 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.761454 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e367f06-ed74-4810-9703-a762eeaae827-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.808862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.808800 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:32:14.815045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.815021 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjdn\" (UniqueName: \"kubernetes.io/projected/2e367f06-ed74-4810-9703-a762eeaae827-kube-api-access-bvjdn\") pod \"kuadrant-console-plugin-66c4fb575d-s8qmv\" (UID: \"2e367f06-ed74-4810-9703-a762eeaae827\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:14.905265 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:14.905238 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" Oct 31 16:32:15.282853 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:15.282806 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv"] Oct 31 16:32:15.283856 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:32:15.283807 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e367f06_ed74_4810_9703_a762eeaae827.slice/crio-5f55f46b42dc087d5437292ad0c734107e77e659c5bfdc5e0348e6c8090e7c8d WatchSource:0}: Error finding container 5f55f46b42dc087d5437292ad0c734107e77e659c5bfdc5e0348e6c8090e7c8d: Status 404 returned error can't find the container with id 5f55f46b42dc087d5437292ad0c734107e77e659c5bfdc5e0348e6c8090e7c8d Oct 31 16:32:15.285861 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:15.285835 4865 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:32:15.938402 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:15.938356 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" event={"ID":"2e367f06-ed74-4810-9703-a762eeaae827","Type":"ContainerStarted","Data":"5f55f46b42dc087d5437292ad0c734107e77e659c5bfdc5e0348e6c8090e7c8d"} Oct 31 16:32:17.684936 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:32:17.683262 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:32:20.227278 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.227239 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw"] Oct 31 16:32:20.231443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.231418 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.284436 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.284409 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw"] Oct 31 16:32:20.323684 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.323657 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.323834 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.323804 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvnm\" (UniqueName: \"kubernetes.io/projected/ea6541d3-4114-4488-a8d1-554f2cfd4805-kube-api-access-2jvnm\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.323906 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.323843 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.330320 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.330293 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:32:20.330535 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.330317 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:32:20.330535 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.330348 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:32:20.332619 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.332599 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:32:20.424552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.424518 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvnm\" (UniqueName: \"kubernetes.io/projected/ea6541d3-4114-4488-a8d1-554f2cfd4805-kube-api-access-2jvnm\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.424721 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.424573 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.424721 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.424627 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.425323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.425299 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.427434 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.427409 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.481136 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.481085 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvnm\" (UniqueName: \"kubernetes.io/projected/ea6541d3-4114-4488-a8d1-554f2cfd4805-kube-api-access-2jvnm\") pod \"maas-api-revoke-sub-d576e68b-l7gtw\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.547292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.547266 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:20.919862 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:20.919786 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw"] Oct 31 16:32:20.923508 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:32:20.923455 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6541d3_4114_4488_a8d1_554f2cfd4805.slice/crio-955d67187f2b2add9d31de4e7f3120164a30201801feb0f00263cd8e1b2fe31b WatchSource:0}: Error finding container 955d67187f2b2add9d31de4e7f3120164a30201801feb0f00263cd8e1b2fe31b: Status 404 returned error can't find the container with id 955d67187f2b2add9d31de4e7f3120164a30201801feb0f00263cd8e1b2fe31b Oct 31 16:32:21.050213 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:21.050174 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" event={"ID":"ea6541d3-4114-4488-a8d1-554f2cfd4805","Type":"ContainerStarted","Data":"955d67187f2b2add9d31de4e7f3120164a30201801feb0f00263cd8e1b2fe31b"} Oct 31 16:32:21.748542 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:21.748507 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:22.058737 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:22.058658 4865 generic.go:356] "Generic (PLEG): container finished" podID="ea6541d3-4114-4488-a8d1-554f2cfd4805" containerID="8006b36dc848e7b6a5e56e11658bc02db634dd83b70c87f5b8a1bde3ab36b157" exitCode=0 Oct 31 16:32:22.058909 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:22.058742 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" event={"ID":"ea6541d3-4114-4488-a8d1-554f2cfd4805","Type":"ContainerDied","Data":"8006b36dc848e7b6a5e56e11658bc02db634dd83b70c87f5b8a1bde3ab36b157"} Oct 31 16:32:23.143880 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.142032 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:23.213822 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.213797 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:23.254446 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.254404 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-xks-ca\") pod \"ea6541d3-4114-4488-a8d1-554f2cfd4805\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " Oct 31 16:32:23.254649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.254579 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-openshift-ca\") pod \"ea6541d3-4114-4488-a8d1-554f2cfd4805\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " Oct 31 16:32:23.254649 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.254640 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6541d3-4114-4488-a8d1-554f2cfd4805-kube-api-access-2jvnm\" (UniqueName: \"kubernetes.io/projected/ea6541d3-4114-4488-a8d1-554f2cfd4805-kube-api-access-2jvnm\") pod \"ea6541d3-4114-4488-a8d1-554f2cfd4805\" (UID: \"ea6541d3-4114-4488-a8d1-554f2cfd4805\") " Oct 31 16:32:23.255025 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.254995 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-openshift-ca" pod "ea6541d3-4114-4488-a8d1-554f2cfd4805" (UID: "ea6541d3-4114-4488-a8d1-554f2cfd4805"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:32:23.257961 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.257937 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-xks-ca" pod "ea6541d3-4114-4488-a8d1-554f2cfd4805" (UID: "ea6541d3-4114-4488-a8d1-554f2cfd4805"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:32:23.257961 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.257950 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6541d3-4114-4488-a8d1-554f2cfd4805-kube-api-access-2jvnm" pod "ea6541d3-4114-4488-a8d1-554f2cfd4805" (UID: "ea6541d3-4114-4488-a8d1-554f2cfd4805"). InnerVolumeSpecName "kube-api-access-2jvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:32:23.355915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.355839 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:32:23.355915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.355879 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6541d3-4114-4488-a8d1-554f2cfd4805-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:32:23.355915 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:23.355897 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvnm\" (UniqueName: \"kubernetes.io/projected/ea6541d3-4114-4488-a8d1-554f2cfd4805-kube-api-access-2jvnm\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:32:24.071127 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:24.071088 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" event={"ID":"2e367f06-ed74-4810-9703-a762eeaae827","Type":"ContainerStarted","Data":"bc826ddd8959f29f24e1323d505b1c14b13aeb4f29bb31c9e7689ad74375dcf5"} Oct 31 16:32:24.072368 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:24.072341 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" event={"ID":"ea6541d3-4114-4488-a8d1-554f2cfd4805","Type":"ContainerDied","Data":"955d67187f2b2add9d31de4e7f3120164a30201801feb0f00263cd8e1b2fe31b"} Oct 31 16:32:24.072529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:24.072370 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw" Oct 31 16:32:24.072529 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:24.072421 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="955d67187f2b2add9d31de4e7f3120164a30201801feb0f00263cd8e1b2fe31b" Oct 31 16:32:24.185779 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:24.185722 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-s8qmv" podStartSLOduration=1.795266313 podStartE2EDuration="10.185708976s" totalImagesPullingTime="8.390442663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:24.182527629 +0000 UTC m=+6389.802079037" watchObservedRunningTime="2026-10-31 16:32:24.185708976 +0000 UTC m=+6389.805260386" Oct 31 16:32:24.529620 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:24.529589 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:26.045003 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:26.044969 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk"] Oct 31 16:32:26.094323 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:26.094288 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5qfsk"] Oct 31 16:32:26.494552 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:26.494524 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9503c8-0823-4bf9-a1e8-f070d56b4e79" path="/var/lib/kubelet/pods/6e9503c8-0823-4bf9-a1e8-f070d56b4e79/volumes" Oct 31 16:32:33.005040 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:33.005006 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:32:37.045332 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:37.045298 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn"] Oct 31 16:32:37.094741 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:37.094708 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mzmmn"] Oct 31 16:32:38.498909 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:38.498858 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949adb0d-c6d2-48d6-a8ea-aa8be62b184e" path="/var/lib/kubelet/pods/949adb0d-c6d2-48d6-a8ea-aa8be62b184e/volumes" Oct 31 16:32:41.646495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:41.645588 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:32:41.646495 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:41.645652 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:32:42.421521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.421488 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:32:42.612598 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.612562 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq"] Oct 31 16:32:42.612968 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.612955 4865 cpu_manager.go:414] "containerMap: removing container" podUID="ea6541d3-4114-4488-a8d1-554f2cfd4805" containerName="revoke-keys" Oct 31 16:32:42.613017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.612970 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6541d3-4114-4488-a8d1-554f2cfd4805" containerName="revoke-keys" Oct 31 16:32:42.613069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.613060 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6541d3-4114-4488-a8d1-554f2cfd4805" containerName="revoke-keys" Oct 31 16:32:42.617385 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.617366 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.671810 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.671750 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq"] Oct 31 16:32:42.710363 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.710322 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.710551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.710395 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmmq9\" (UniqueName: \"kubernetes.io/projected/9547c5c8-a423-4002-870d-fd1648576ad7-kube-api-access-fmmq9\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.710551 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.710439 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.717811 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.717770 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:32:42.717969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.717863 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:32:42.717969 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.717951 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:32:42.719113 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.719091 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:32:42.811299 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.811265 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.811454 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.811315 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmmq9\" (UniqueName: \"kubernetes.io/projected/9547c5c8-a423-4002-870d-fd1648576ad7-kube-api-access-fmmq9\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.811454 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.811353 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.811953 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.811934 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.813875 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.813855 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.867696 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.867672 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmmq9\" (UniqueName: \"kubernetes.io/projected/9547c5c8-a423-4002-870d-fd1648576ad7-kube-api-access-fmmq9\") pod \"maas-api-revoke-sub-d2bc93bc-sb9dq\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:42.930549 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:42.930488 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:43.300002 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:43.299975 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq"] Oct 31 16:32:43.300870 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:32:43.300844 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9547c5c8_a423_4002_870d_fd1648576ad7.slice/crio-2c0ba74dbdb72b10d44d2a71c39fda038c177aac23c6d35395674416b3f83d51 WatchSource:0}: Error finding container 2c0ba74dbdb72b10d44d2a71c39fda038c177aac23c6d35395674416b3f83d51: Status 404 returned error can't find the container with id 2c0ba74dbdb72b10d44d2a71c39fda038c177aac23c6d35395674416b3f83d51 Oct 31 16:32:44.247428 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:44.247394 4865 generic.go:356] "Generic (PLEG): container finished" podID="9547c5c8-a423-4002-870d-fd1648576ad7" containerID="66e862191fddc8a523a66d6c410fb8ffc8a7cbe7db90f44d7ffbd8aa60e74e4b" exitCode=0 Oct 31 16:32:44.247859 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:44.247498 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" event={"ID":"9547c5c8-a423-4002-870d-fd1648576ad7","Type":"ContainerDied","Data":"66e862191fddc8a523a66d6c410fb8ffc8a7cbe7db90f44d7ffbd8aa60e74e4b"} Oct 31 16:32:44.247859 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:44.247549 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" event={"ID":"9547c5c8-a423-4002-870d-fd1648576ad7","Type":"ContainerStarted","Data":"2c0ba74dbdb72b10d44d2a71c39fda038c177aac23c6d35395674416b3f83d51"} Oct 31 16:32:45.387567 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.387543 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:45.540263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.540183 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-xks-ca\") pod \"9547c5c8-a423-4002-870d-fd1648576ad7\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " Oct 31 16:32:45.540458 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.540284 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-openshift-ca\") pod \"9547c5c8-a423-4002-870d-fd1648576ad7\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " Oct 31 16:32:45.540458 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.540368 4865 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9547c5c8-a423-4002-870d-fd1648576ad7-kube-api-access-fmmq9\" (UniqueName: \"kubernetes.io/projected/9547c5c8-a423-4002-870d-fd1648576ad7-kube-api-access-fmmq9\") pod \"9547c5c8-a423-4002-870d-fd1648576ad7\" (UID: \"9547c5c8-a423-4002-870d-fd1648576ad7\") " Oct 31 16:32:45.540645 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.540620 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-openshift-ca" pod "9547c5c8-a423-4002-870d-fd1648576ad7" (UID: "9547c5c8-a423-4002-870d-fd1648576ad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:32:45.540754 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.540737 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-openshift-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:32:45.542632 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.542606 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9547c5c8-a423-4002-870d-fd1648576ad7-kube-api-access-fmmq9" pod "9547c5c8-a423-4002-870d-fd1648576ad7" (UID: "9547c5c8-a423-4002-870d-fd1648576ad7"). InnerVolumeSpecName "kube-api-access-fmmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:32:45.542728 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.542630 4865 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-xks-ca" pod "9547c5c8-a423-4002-870d-fd1648576ad7" (UID: "9547c5c8-a423-4002-870d-fd1648576ad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:32:45.641679 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.641644 4865 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmmq9\" (UniqueName: \"kubernetes.io/projected/9547c5c8-a423-4002-870d-fd1648576ad7-kube-api-access-fmmq9\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:32:45.641679 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:45.641679 4865 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9547c5c8-a423-4002-870d-fd1648576ad7-maas-api-xks-ca\") on node \"ip-10-0-131-60.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:32:46.262421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:46.262381 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" event={"ID":"9547c5c8-a423-4002-870d-fd1648576ad7","Type":"ContainerDied","Data":"2c0ba74dbdb72b10d44d2a71c39fda038c177aac23c6d35395674416b3f83d51"} Oct 31 16:32:46.262421 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:46.262420 4865 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq" Oct 31 16:32:46.262658 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:46.262444 4865 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0ba74dbdb72b10d44d2a71c39fda038c177aac23c6d35395674416b3f83d51" Oct 31 16:32:47.488452 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:32:47.488418 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:32:50.525123 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:50.525078 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:32:55.625052 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.625020 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7468745db8-rgwh9"] Oct 31 16:32:55.625460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.625420 4865 cpu_manager.go:414] "containerMap: removing container" podUID="9547c5c8-a423-4002-870d-fd1648576ad7" containerName="revoke-keys" Oct 31 16:32:55.625460 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.625431 4865 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9547c5c8-a423-4002-870d-fd1648576ad7" containerName="revoke-keys" Oct 31 16:32:55.625560 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.625520 4865 memory_manager.go:365] "RemoveStaleState removing state" podUID="9547c5c8-a423-4002-870d-fd1648576ad7" containerName="revoke-keys" Oct 31 16:32:55.640906 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.640879 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.692870 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.692830 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:32:55.733397 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.733367 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878e8da-2005-4bb9-a74d-6de65d44f224-openshift-service-ca\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.733582 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.733415 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878e8da-2005-4bb9-a74d-6de65d44f224-metrics-certs\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.733582 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.733540 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvqz\" (UniqueName: \"kubernetes.io/projected/7878e8da-2005-4bb9-a74d-6de65d44f224-kube-api-access-wgvqz\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.733702 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.733595 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878e8da-2005-4bb9-a74d-6de65d44f224-maas-api-tls\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.742501 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.742478 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:32:55.742787 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.742772 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 16:32:55.742787 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.742769 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 16:32:55.743017 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.742861 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\"" Oct 31 16:32:55.746372 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.746351 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7468745db8-rgwh9"] Oct 31 16:32:55.834427 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.834403 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878e8da-2005-4bb9-a74d-6de65d44f224-metrics-certs\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.834597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.834460 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvqz\" (UniqueName: \"kubernetes.io/projected/7878e8da-2005-4bb9-a74d-6de65d44f224-kube-api-access-wgvqz\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.834597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.834524 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878e8da-2005-4bb9-a74d-6de65d44f224-maas-api-tls\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.834597 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.834585 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878e8da-2005-4bb9-a74d-6de65d44f224-openshift-service-ca\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.835263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.835180 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7878e8da-2005-4bb9-a74d-6de65d44f224-openshift-service-ca\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.837026 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.837002 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7878e8da-2005-4bb9-a74d-6de65d44f224-maas-api-tls\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.837131 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.837046 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7878e8da-2005-4bb9-a74d-6de65d44f224-metrics-certs\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.890628 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.890572 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvqz\" (UniqueName: \"kubernetes.io/projected/7878e8da-2005-4bb9-a74d-6de65d44f224-kube-api-access-wgvqz\") pod \"maas-api-7468745db8-rgwh9\" (UID: \"7878e8da-2005-4bb9-a74d-6de65d44f224\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:55.955133 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:55.955096 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:56.324944 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:56.324922 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7468745db8-rgwh9"] Oct 31 16:32:56.326159 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:32:56.326137 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7878e8da_2005_4bb9_a74d_6de65d44f224.slice/crio-0742843aedeb3a7825d6611e19dc9ee3b8e7431a31734afeaf6738cf5fe62eeb WatchSource:0}: Error finding container 0742843aedeb3a7825d6611e19dc9ee3b8e7431a31734afeaf6738cf5fe62eeb: Status 404 returned error can't find the container with id 0742843aedeb3a7825d6611e19dc9ee3b8e7431a31734afeaf6738cf5fe62eeb Oct 31 16:32:57.327950 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:57.327914 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" event={"ID":"7878e8da-2005-4bb9-a74d-6de65d44f224","Type":"ContainerStarted","Data":"0742843aedeb3a7825d6611e19dc9ee3b8e7431a31734afeaf6738cf5fe62eeb"} Oct 31 16:32:59.340000 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:59.339962 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" event={"ID":"7878e8da-2005-4bb9-a74d-6de65d44f224","Type":"ContainerStarted","Data":"a79dc054f9d13f78c69bcbb92784da21f5430bf00ec90bdf3f44aac0feb4db7c"} Oct 31 16:32:59.340545 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:59.340091 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:32:59.454780 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:32:59.454727 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" podStartSLOduration=2.420703368 podStartE2EDuration="4.454713236s" totalImagesPullingTime="2.034009868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:59.453071496 +0000 UTC m=+6425.072622907" watchObservedRunningTime="2026-10-31 16:32:59.454713236 +0000 UTC m=+6425.074264646" Oct 31 16:33:01.500018 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:01.499984 4865 scope.go:122] "RemoveContainer" containerID="9373deaded8d1f439c4199480bd2d557ade09ce3545a8a97cd5597bd40a65a6c" Oct 31 16:33:01.508916 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:01.508893 4865 scope.go:122] "RemoveContainer" containerID="241f24ecd4e22fb51d83e30cca0461137e71e58a40e0e90c043bbc5f7a04a724" Oct 31 16:33:01.516826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:01.516804 4865 scope.go:122] "RemoveContainer" containerID="e00b0249f64bdc469ee73d04088cf0437b6e78284378b72845a99386e6852344" Oct 31 16:33:05.351708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:05.351634 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7468745db8-rgwh9" Oct 31 16:33:08.127155 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:08.127122 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:33:09.165502 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:09.165442 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:09.287329 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:09.287286 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:11.663494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:11.662584 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:33:11.663494 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:11.662672 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:33:17.488793 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:33:17.488755 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:33:22.072706 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:22.072675 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:23.492084 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:23.492046 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:24.890753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:24.890718 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:44.821457 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:44.821422 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:33:47.490323 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:33:47.490288 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:33:48.754024 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:48.753986 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:53.671520 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:53.670541 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:33:53.671520 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:53.670614 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:33:56.932691 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:33:56.932650 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:34:01.544735 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:01.544702 4865 scope.go:122] "RemoveContainer" containerID="09fd8a4cd3d9520dac32431eab587ce1a18331a3e95b082a3f09e1adbb378859" Oct 31 16:34:01.553774 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:01.553754 4865 scope.go:122] "RemoveContainer" containerID="cc16386a104b7d46d6e25c567709d1c8c4509d3c3739f421e76b63b18768650e" Oct 31 16:34:01.562248 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:01.562228 4865 scope.go:122] "RemoveContainer" containerID="b5f7b39388185f5f0f1db239aab18e7be6bb343f14caebacf8fa41a7d851f156" Oct 31 16:34:13.055893 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:13.055853 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:17.490600 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:34:17.490570 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:34:18.194317 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:18.194285 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:34:22.382993 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:22.382956 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:23.895593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:23.895560 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:25.217140 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:25.217102 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:28.053586 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:28.053539 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn"] Oct 31 16:34:28.108230 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:28.108187 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e55dc89-jz4xn"] Oct 31 16:34:28.500352 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:28.500310 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a474017b-6bb5-4348-ac26-af537e6a6dfe" path="/var/lib/kubelet/pods/a474017b-6bb5-4348-ac26-af537e6a6dfe/volumes" Oct 31 16:34:30.301387 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:30.301342 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:34:47.490939 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:34:47.490899 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:34:50.713117 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:34:50.713083 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:01.009030 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:01.008993 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:35:11.688335 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:11.688300 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:35:17.489544 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:35:17.489502 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:35:22.739629 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:22.739590 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:24.144775 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:24.144737 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:35:24.176827 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:24.176802 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:25.565550 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:25.565507 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:34.044599 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:34.044565 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth"] Oct 31 16:35:34.093839 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:34.093807 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2hsth"] Oct 31 16:35:34.495657 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:34.495628 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d37b74f-2921-41c5-9468-5b2e6824cbef" path="/var/lib/kubelet/pods/8d37b74f-2921-41c5-9468-5b2e6824cbef/volumes" Oct 31 16:35:36.438826 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:36.438790 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:35:47.489951 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:35:47.489911 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:35:56.546636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:56.546609 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:35:56.555123 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:56.555103 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:35:58.932680 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:58.932651 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gdnkp_1d447892-64c2-41a6-a155-65d0717627fd/console-operator/2.log" Oct 31 16:35:58.950464 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:35:58.950434 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4knfg_f1152975-ff11-49bf-b390-022434aae2a0/ovn-acl-logging/0.log" Oct 31 16:36:00.008827 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:00.008793 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:36:01.608493 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:01.599696 4865 scope.go:122] "RemoveContainer" containerID="a7e9fca9b33d3fb657113e4b97f604327b7df35c6b053efbf56ae11c80dd580f" Oct 31 16:36:10.642560 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:10.642484 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:36:17.589550 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:36:17.589462 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:36:20.209874 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:20.209837 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:36:23.097263 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:23.097226 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:25.538504 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:25.538457 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:40.513394 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:40.513357 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:36:47.490094 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:36:47.490057 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:36:49.939980 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:36:49.939937 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:37:01.801708 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:01.801676 4865 scope.go:122] "RemoveContainer" containerID="df5f2feed189a6f478a741026a71ccacdff9e3467e60c53152e4e4c1c94e9ffa" Oct 31 16:37:17.489052 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:37:17.489013 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:37:23.483030 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:23.482994 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:24.045891 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:24.045859 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw"] Oct 31 16:37:24.095542 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:24.095511 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-l7gtw"] Oct 31 16:37:24.495636 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:24.495602 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6541d3-4114-4488-a8d1-554f2cfd4805" path="/var/lib/kubelet/pods/ea6541d3-4114-4488-a8d1-554f2cfd4805/volumes" Oct 31 16:37:25.900334 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:25.900300 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:26.278163 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:26.278124 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:35.026638 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:35.026596 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:37:35.027338 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:35.027289 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:37:46.065796 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:46.065757 4865 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq"] Oct 31 16:37:46.124533 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:46.121859 4865 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sb9dq"] Oct 31 16:37:46.509498 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:46.504152 4865 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9547c5c8-a423-4002-870d-fd1648576ad7" path="/var/lib/kubelet/pods/9547c5c8-a423-4002-870d-fd1648576ad7/volumes" Oct 31 16:37:47.489001 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:37:47.488964 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:37:56.657676 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:56.657634 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:37:56.658524 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:37:56.658489 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwld2" podUID="1b390bce-baa8-4fdd-895f-c660aafd1813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:38:08.298131 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:08.297581 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-r24tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:38:08.298131 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:08.297667 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-r24tk" podUID="97891b87-02cb-4007-beed-5903c97d4380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:38:08.685652 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:08.685552 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-l6znw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:38:08.686404 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:08.685929 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-l6znw" podUID="7ed4866e-493a-4027-9fa6-5d4858002f16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:38:17.599787 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:38:17.599748 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:38:23.758926 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:23.758891 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:26.244036 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:26.244004 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:47.490521 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:38:47.490461 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:38:50.026629 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:50.026583 4865 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bgh7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:38:50.027325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:50.027291 4865 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bgh7p" podUID="0d25046d-9187-4e7c-b413-612b4446cb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:38:58.503463 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:38:58.503426 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2f7n9"] Oct 31 16:39:01.830683 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:01.830653 4865 scope.go:122] "RemoveContainer" containerID="66e862191fddc8a523a66d6c410fb8ffc8a7cbe7db90f44d7ffbd8aa60e74e4b" Oct 31 16:39:01.839409 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:01.839380 4865 scope.go:122] "RemoveContainer" containerID="8006b36dc848e7b6a5e56e11658bc02db634dd83b70c87f5b8a1bde3ab36b157" Oct 31 16:39:17.489823 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:39:17.489729 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:39:25.405692 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:25.405653 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:26.593134 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:26.593105 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:27.561292 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:27.561262 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz97q_200047c9-fccd-4826-b521-fce830acd38f/manager/0.log" Oct 31 16:39:28.771880 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:28.771843 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-dff98979c-mrlql_83c1b19d-24d1-429b-ab1a-955010b8c9d4/manager/0.log" Oct 31 16:39:29.565032 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:29.565001 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5vzk_4e96a5f7-706c-4405-be53-e4362dc7677d/manager/0.log" Oct 31 16:39:31.464759 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:31.464725 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7468745db8-rgwh9_7878e8da-2005-4bb9-a74d-6de65d44f224/maas-api/0.log" Oct 31 16:39:38.140308 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:38.140269 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s8qmv_2e367f06-ed74-4810-9703-a762eeaae827/kuadrant-console-plugin/0.log" Oct 31 16:39:38.936185 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:38.936151 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2f7n9_cd1efbf8-d417-453e-a9e4-0be4b5b6d393/limitador/0.log" Oct 31 16:39:41.130398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:41.130362 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d7844885-gnrpl_3c7543a2-6d1f-4a11-821c-2cd20444239e/kube-auth-proxy/0.log" Oct 31 16:39:41.566909 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:41.566850 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-bgh7p_0d25046d-9187-4e7c-b413-612b4446cb3a/istio-proxy/0.log" Oct 31 16:39:42.334069 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:42.334023 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-l6znw_7ed4866e-493a-4027-9fa6-5d4858002f16/istio-proxy/0.log" Oct 31 16:39:42.893104 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:42.893077 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-pwld2_1b390bce-baa8-4fdd-895f-c660aafd1813/istio-proxy/0.log" Oct 31 16:39:43.163736 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:43.163635 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-r24tk_97891b87-02cb-4007-beed-5903c97d4380/istio-proxy/0.log" Oct 31 16:39:44.352168 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:44.352125 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b99d75bdc-tcjgn_47f2d453-01ec-48cb-aa01-ba04ef0dbfc8/payload-pre-processing/0.log" Oct 31 16:39:44.556450 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:44.556420 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7744dd7796-82xn4_bbfa389a-4e98-487b-ab65-574e7c00176a/payload-processing/0.log" Oct 31 16:39:44.787246 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:44.787210 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-699cdb64d9-r57rx_e3ce5b4f-032b-4791-bc7d-0e5916aba070/router/0.log" Oct 31 16:39:46.385693 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:46.385636 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-22b9j_4ded9cdd-cf9d-4bf8-a6aa-e519619b8ea8/main/0.log" Oct 31 16:39:46.838801 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:46.838768 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-r6fn8_7b8fa3fc-a649-4b87-b616-4156319382e3/main/0.log" Oct 31 16:39:47.291819 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:47.291791 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2fbkj_1fd003d5-b7f4-42bf-87ef-bfa1a035d906/main/0.log" Oct 31 16:39:47.489721 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:39:47.489683 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:39:47.744583 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:47.744551 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lbrhb_a447a957-8083-43b4-974e-0cc8f358cdba/main/0.log" Oct 31 16:39:58.342793 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.342762 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kt42p/must-gather-nxqtl"] Oct 31 16:39:58.346399 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.346381 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.354325 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.354302 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:58.402798 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.402768 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt42p/must-gather-nxqtl"] Oct 31 16:39:58.448604 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.448570 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt42p\"/\"openshift-service-ca.crt\"" Oct 31 16:39:58.448604 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.448577 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt42p\"/\"kube-root-ca.crt\"" Oct 31 16:39:58.449395 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.449376 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kt42p\"/\"default-dockercfg-9swp7\"" Oct 31 16:39:58.458965 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.458943 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b9r2f/must-gather-z6wz5"] Oct 31 16:39:58.459235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.459215 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjzx\" (UniqueName: \"kubernetes.io/projected/63bc6204-afd7-41c0-a42c-869050b6d8f9-kube-api-access-nzjzx\") pod \"must-gather-nxqtl\" (UID: \"63bc6204-afd7-41c0-a42c-869050b6d8f9\") " pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.459339 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.459320 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63bc6204-afd7-41c0-a42c-869050b6d8f9-must-gather-output\") pod \"must-gather-nxqtl\" (UID: \"63bc6204-afd7-41c0-a42c-869050b6d8f9\") " pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.462790 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.462776 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:58.514835 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.514809 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9r2f\"/\"kube-root-ca.crt\"" Oct 31 16:39:58.514835 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.514827 4865 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9r2f\"/\"openshift-service-ca.crt\"" Oct 31 16:39:58.515088 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.514908 4865 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b9r2f\"/\"default-dockercfg-ztmvf\"" Oct 31 16:39:58.517235 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.517210 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/must-gather-z6wz5"] Oct 31 16:39:58.560199 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.560171 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63bc6204-afd7-41c0-a42c-869050b6d8f9-must-gather-output\") pod \"must-gather-nxqtl\" (UID: \"63bc6204-afd7-41c0-a42c-869050b6d8f9\") " pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.560443 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.560425 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjzx\" (UniqueName: \"kubernetes.io/projected/63bc6204-afd7-41c0-a42c-869050b6d8f9-kube-api-access-nzjzx\") pod \"must-gather-nxqtl\" (UID: \"63bc6204-afd7-41c0-a42c-869050b6d8f9\") " pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.560564 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.560453 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/63bc6204-afd7-41c0-a42c-869050b6d8f9-must-gather-output\") pod \"must-gather-nxqtl\" (UID: \"63bc6204-afd7-41c0-a42c-869050b6d8f9\") " pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.603990 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.603931 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:58.616712 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.616685 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjzx\" (UniqueName: \"kubernetes.io/projected/63bc6204-afd7-41c0-a42c-869050b6d8f9-kube-api-access-nzjzx\") pod \"must-gather-nxqtl\" (UID: \"63bc6204-afd7-41c0-a42c-869050b6d8f9\") " pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.660398 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.660377 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt42p/must-gather-nxqtl" Oct 31 16:39:58.661791 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.661771 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5eab280d-c1c4-4651-b21d-3fbfeac903a4-must-gather-output\") pod \"must-gather-z6wz5\" (UID: \"5eab280d-c1c4-4651-b21d-3fbfeac903a4\") " pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:58.661896 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.661884 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbv6n\" (UniqueName: \"kubernetes.io/projected/5eab280d-c1c4-4651-b21d-3fbfeac903a4-kube-api-access-sbv6n\") pod \"must-gather-z6wz5\" (UID: \"5eab280d-c1c4-4651-b21d-3fbfeac903a4\") " pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:58.762441 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.762412 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5eab280d-c1c4-4651-b21d-3fbfeac903a4-must-gather-output\") pod \"must-gather-z6wz5\" (UID: \"5eab280d-c1c4-4651-b21d-3fbfeac903a4\") " pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:58.762654 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.762518 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbv6n\" (UniqueName: \"kubernetes.io/projected/5eab280d-c1c4-4651-b21d-3fbfeac903a4-kube-api-access-sbv6n\") pod \"must-gather-z6wz5\" (UID: \"5eab280d-c1c4-4651-b21d-3fbfeac903a4\") " pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:58.762841 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.762818 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5eab280d-c1c4-4651-b21d-3fbfeac903a4-must-gather-output\") pod \"must-gather-z6wz5\" (UID: \"5eab280d-c1c4-4651-b21d-3fbfeac903a4\") " pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:58.822346 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:58.822310 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbv6n\" (UniqueName: \"kubernetes.io/projected/5eab280d-c1c4-4651-b21d-3fbfeac903a4-kube-api-access-sbv6n\") pod \"must-gather-z6wz5\" (UID: \"5eab280d-c1c4-4651-b21d-3fbfeac903a4\") " pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:59.030581 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:59.030546 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt42p/must-gather-nxqtl"] Oct 31 16:39:59.031999 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:39:59.031971 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63bc6204_afd7_41c0_a42c_869050b6d8f9.slice/crio-099e1f4e8bc1125df09237a138dce7c3fa50694dcd7a3fee2af539f2cc970f8f WatchSource:0}: Error finding container 099e1f4e8bc1125df09237a138dce7c3fa50694dcd7a3fee2af539f2cc970f8f: Status 404 returned error can't find the container with id 099e1f4e8bc1125df09237a138dce7c3fa50694dcd7a3fee2af539f2cc970f8f Oct 31 16:39:59.034184 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:59.034167 4865 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:39:59.076266 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:59.076237 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/must-gather-z6wz5" Oct 31 16:39:59.444045 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:59.443948 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt42p/must-gather-nxqtl" event={"ID":"63bc6204-afd7-41c0-a42c-869050b6d8f9","Type":"ContainerStarted","Data":"099e1f4e8bc1125df09237a138dce7c3fa50694dcd7a3fee2af539f2cc970f8f"} Oct 31 16:39:59.445924 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:39:59.445902 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/must-gather-z6wz5"] Oct 31 16:39:59.447185 ip-10-0-131-60 kubenswrapper[4865]: W1031 16:39:59.447159 4865 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5eab280d_c1c4_4651_b21d_3fbfeac903a4.slice/crio-c9b11e253eb4f829164276db6813860af40676fa7fdb3a77816c5b25bb93dc87 WatchSource:0}: Error finding container c9b11e253eb4f829164276db6813860af40676fa7fdb3a77816c5b25bb93dc87: Status 404 returned error can't find the container with id c9b11e253eb4f829164276db6813860af40676fa7fdb3a77816c5b25bb93dc87 Oct 31 16:40:00.451521 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:00.451450 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/must-gather-z6wz5" event={"ID":"5eab280d-c1c4-4651-b21d-3fbfeac903a4","Type":"ContainerStarted","Data":"c9b11e253eb4f829164276db6813860af40676fa7fdb3a77816c5b25bb93dc87"} Oct 31 16:40:01.460139 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:01.460105 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/must-gather-z6wz5" event={"ID":"5eab280d-c1c4-4651-b21d-3fbfeac903a4","Type":"ContainerStarted","Data":"d3d9d8cf131ec0c61194b97bf205ad5156d3a2d3bcd5e098c0b7ba3ab9acc20a"} Oct 31 16:40:01.460576 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:01.460148 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/must-gather-z6wz5" event={"ID":"5eab280d-c1c4-4651-b21d-3fbfeac903a4","Type":"ContainerStarted","Data":"3e7f6ca7f3990f09169ec83eaf15836e7545b01cc78b15c3fff0694e16fdb215"} Oct 31 16:40:01.571358 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:01.571288 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b9r2f/must-gather-z6wz5" podStartSLOduration=2.331339895 podStartE2EDuration="3.571269143s" totalImagesPullingTime="1.239929248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:01.570609598 +0000 UTC m=+6847.190161009" watchObservedRunningTime="2026-10-31 16:40:01.571269143 +0000 UTC m=+6847.190820555" Oct 31 16:40:06.496039 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:06.496004 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt42p/must-gather-nxqtl" event={"ID":"63bc6204-afd7-41c0-a42c-869050b6d8f9","Type":"ContainerStarted","Data":"2ac432a23bd4210b81a5444adef9f788865d444a99e47ad8bff7b3a0771eee14"} Oct 31 16:40:06.496433 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:06.496047 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt42p/must-gather-nxqtl" event={"ID":"63bc6204-afd7-41c0-a42c-869050b6d8f9","Type":"ContainerStarted","Data":"30d5b65a4ffd278d9948ecf359745755b6634a40fb27a6e52f8ab8520eaba224"} Oct 31 16:40:06.607111 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:06.607003 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kt42p/must-gather-nxqtl" podStartSLOduration=1.387349717 podStartE2EDuration="8.606988364s" totalImagesPullingTime="7.219638647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:06.605136788 +0000 UTC m=+6852.224688202" watchObservedRunningTime="2026-10-31 16:40:06.606988364 +0000 UTC m=+6852.226539786" Oct 31 16:40:08.736074 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:08.735873 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h78q8_8eefaca9-ad88-44ac-be6b-363497696b72/global-pull-secret-syncer/0.log" Oct 31 16:40:09.903654 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:09.903622 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5p7xg_14236efa-49b0-452b-8136-157cdb6390cf/connection-checker/0.log" Oct 31 16:40:10.330664 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:10.329804 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-b4frh_45be3fe3-424d-4775-8203-ec94e0853bc1/connection-checker/0.log" Oct 31 16:40:10.767226 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:10.767190 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gq2qd_8bffb79a-a3c4-4fc8-b742-148342ad6c35/connection-checker/0.log" Oct 31 16:40:11.271247 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:11.271213 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" Oct 31 16:40:11.742255 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:11.742191 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz97q_200047c9-fccd-4826-b521-fce830acd38f/manager/0.log" Oct 31 16:40:12.612651 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:12.612615 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-60.us-west-1.compute.internal_71acc277a69aa4c1becf340e4de5eca3/haproxy/0.log" Oct 31 16:40:12.852073 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:12.852029 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-dff98979c-mrlql_83c1b19d-24d1-429b-ab1a-955010b8c9d4/manager/0.log" Oct 31 16:40:14.698748 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:14.698710 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5vzk_4e96a5f7-706c-4405-be53-e4362dc7677d/manager/0.log" Oct 31 16:40:17.513491 ip-10-0-131-60 kubenswrapper[4865]: E1031 16:40:17.513266 4865 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-clmnr" containerName="extractor" Oct 31 16:40:18.439818 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:18.439774 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz97q_200047c9-fccd-4826-b521-fce830acd38f/manager/0.log" Oct 31 16:40:19.341674 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.341635 4865 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6"] Oct 31 16:40:19.345562 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.345536 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.399411 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.399377 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6"] Oct 31 16:40:19.513927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.513889 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-sys\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.513927 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.513926 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-podres\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.514201 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.514060 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-lib-modules\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.514201 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.514117 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65m5q\" (UniqueName: \"kubernetes.io/projected/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-kube-api-access-65m5q\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.514201 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.514156 4865 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-proc\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.615630 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65m5q\" (UniqueName: \"kubernetes.io/projected/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-kube-api-access-65m5q\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.615730 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-proc\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.615794 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-sys\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.615827 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-podres\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.615927 4865 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-lib-modules\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.616049 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-lib-modules\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.616094 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-sys\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.616121 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-podres\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.616742 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.616386 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-proc\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.675417 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.675357 4865 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65m5q\" (UniqueName: \"kubernetes.io/projected/c55be6ae-fd7f-42e7-957b-7c0bc25e4bca-kube-api-access-65m5q\") pod \"perf-node-gather-daemonset-b8tn6\" (UID: \"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:19.836939 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.835711 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-dff98979c-mrlql_83c1b19d-24d1-429b-ab1a-955010b8c9d4/manager/0.log" Oct 31 16:40:19.962938 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:19.962908 4865 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:20.347725 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:20.347675 4865 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6"] Oct 31 16:40:20.523570 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:20.523539 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s8qmv_2e367f06-ed74-4810-9703-a762eeaae827/kuadrant-console-plugin/0.log" Oct 31 16:40:20.714556 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:20.714510 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" event={"ID":"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca","Type":"ContainerStarted","Data":"88e98114469099b79c35ea2553eebc3f73d4a57523536109fb58f64daf1477e9"} Oct 31 16:40:20.714782 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:20.714630 4865 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" event={"ID":"c55be6ae-fd7f-42e7-957b-7c0bc25e4bca","Type":"ContainerStarted","Data":"9d5b40c100a5161814d35b8fede924a62d43c6e9afa47c13e7e8685a1c75bef0"} Oct 31 16:40:20.715236 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:20.715207 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:20.831689 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:20.831635 4865 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" podStartSLOduration=1.831621073 podStartE2EDuration="1.831621073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:20.828270582 +0000 UTC m=+6866.447821993" watchObservedRunningTime="2026-10-31 16:40:20.831621073 +0000 UTC m=+6866.451172483" Oct 31 16:40:21.373142 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:21.373105 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2f7n9_cd1efbf8-d417-453e-a9e4-0be4b5b6d393/limitador/0.log" Oct 31 16:40:21.563483 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:21.563429 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5vzk_4e96a5f7-706c-4405-be53-e4362dc7677d/manager/0.log" Oct 31 16:40:24.440233 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:24.440195 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:24.876593 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:24.876559 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz97q_200047c9-fccd-4826-b521-fce830acd38f/manager/0.log" Oct 31 16:40:26.132260 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:26.132220 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-dff98979c-mrlql_83c1b19d-24d1-429b-ab1a-955010b8c9d4/manager/0.log" Oct 31 16:40:26.734425 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:26.734386 4865 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-b8tn6" Oct 31 16:40:26.938014 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:26.937961 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:27.291753 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:27.291714 4865 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2bgb4_6f9b42eb-56a8-4c76-9fbe-4d2c21d2e9f4/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:27.597909 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:27.597828 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-b8jpk_76e4782a-cfd3-4b3e-9d37-01c836d0e0ad/cluster-monitoring-operator/0.log" Oct 31 16:40:27.923952 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:27.923848 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5vzk_4e96a5f7-706c-4405-be53-e4362dc7677d/manager/0.log" Oct 31 16:40:31.317569 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:31.317536 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjdc4_c06d05c5-fee2-433e-ab10-c368ebb51711/node-exporter/0.log" Oct 31 16:40:31.733487 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:31.733446 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjdc4_c06d05c5-fee2-433e-ab10-c368ebb51711/kube-rbac-proxy/0.log" Oct 31 16:40:31.912012 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:31.911985 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d7844885-gnrpl_3c7543a2-6d1f-4a11-821c-2cd20444239e/kube-auth-proxy/0.log" Oct 31 16:40:32.150609 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:32.150527 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjdc4_c06d05c5-fee2-433e-ab10-c368ebb51711/init-textfile/0.log" Oct 31 16:40:32.342764 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:32.342731 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-bgh7p_0d25046d-9187-4e7c-b413-612b4446cb3a/istio-proxy/0.log" Oct 31 16:40:32.986124 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:32.985849 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-l6znw_7ed4866e-493a-4027-9fa6-5d4858002f16/istio-proxy/0.log" Oct 31 16:40:33.311953 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:33.311874 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-pwld2_1b390bce-baa8-4fdd-895f-c660aafd1813/istio-proxy/0.log" Oct 31 16:40:33.524710 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:33.524686 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-r24tk_97891b87-02cb-4007-beed-5903c97d4380/istio-proxy/0.log" Oct 31 16:40:34.610632 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:34.610606 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b99d75bdc-tcjgn_47f2d453-01ec-48cb-aa01-ba04ef0dbfc8/payload-pre-processing/0.log" Oct 31 16:40:34.821967 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:34.821938 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7744dd7796-82xn4_bbfa389a-4e98-487b-ab65-574e7c00176a/payload-processing/0.log" Oct 31 16:40:35.043283 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:35.043258 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-699cdb64d9-r57rx_e3ce5b4f-032b-4791-bc7d-0e5916aba070/router/0.log" Oct 31 16:40:41.289620 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:41.289592 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-s8qmv_2e367f06-ed74-4810-9703-a762eeaae827/kuadrant-console-plugin/0.log" Oct 31 16:40:41.811239 ip-10-0-131-60 kubenswrapper[4865]: I1031 16:40:41.811209 4865 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2f7n9_cd1efbf8-d417-453e-a9e4-0be4b5b6d393/limitador/0.log"